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ABSTRACT 


iA  syntactic  method  lor  the  analysis  of  time-varying  image  patterns 
is  proposed  and  studied.  This  method  utilizes  translation  schema  to 
model  the  time-varying  properties  of  image  patterns.  A  syntactic 
deformation  model  is  first  applied  to  transform  the  i-th  image  into  the 
(i+1)-th  image  of  an  image  sequence.  Then  the  concept  of  translation  in 
♦ormal  language  theory  is  used  as  a  mechanism  to  characterize  the 
dynamic  process  of  the  image  sequence.  A  formulation  ot  stochastic 
translation  is  also  presented.  A  generalized  synt ax-d i rec t eU  tree 
translation  model  is  proposed  to  handle  high-dimensional  patterns.  The 
generalized  moael  is  compared  with  the  conventional  top-down  ano 
at  tom-up  tree  translation  models. 

A  traffic  monitoring  problem  is  analyzed  using  the  proposed  tree 

'ranslation  model.  Each  input  image  is  represented  as  a  tree  structure. 

he  proposed  tree  translation  model  is  useo  to  model  the  variation  ot 

mage  content  between  consecutive  images.  A  parsing  algorithm  for  tree 

■ranslation  is  applied  to  match  moving  objects  (vehicles)  in  each  pair 

of  consecutive  images,  j 

\ . 

i 
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CHAPTER  1 

INTRODUCTION 
_1’1  Introduction 

During  the  past  two  decades,  there  has  been  an  increasing  interest 
in  pattern  recognition.  Most  of  the  developments  in  the  theory  and 
applications  of  pattern  recognition  use  the  statistical  approach  LI-63. 
In  order  to  represent  the  structural  information  contained  in  the 
patterns,  the  syntactic  or  structural  approach  has  been  proposed  L7-10J. 
The  precision  of  syntactic  specification  provides  the  recognition 
procedure  not  only  the  capability  of  classifying  patterns  but  also  the 
capacity  of  describing  patterns.  Recently,  the  problem  of  time-varying 
image  analysis  has  drawn  great  attention  C 1 5—20 J .  In  this  research,  this 
problem  will  be  analyzed  using  syntactic  method. 

A  syntactic  pattern  recognition  system  consists  of  3  major  parts: 
(a)  preprocessing  and  segmentat ion,  (b)  primitive  extraction,  ana  (c) 
syntax  analysis.  To  analyze  time-varying  images,  first  of  all,  we  need 
to  select. a  proper  representation  method.  Then  a  syntax  analysis  scheme 
has  to  be  devised. 

In  the  past,  string  grammar  has  been  applied  to  the  problem  of 
shape  analysis  and  waveform  analysis,  tree  grammar  has  been  applied  in 
the  problem  of  texture  analysis,  fingerprint  recognition,  and  scene 
analysis.  What’s  needed  here  is  some  description  method  which  can  be 


applied  to  represent  and  analyze  general  time-varying  image  patterns. 


Survey 

1.2.1  Pattern  recognition 

The  many  different  mathematical  techniques  used  to  solve  pattern 
recognition  problems  may  be  grouped  into  two  general  approaches;  namely, 
the  decision-theoretic  approach  and  the  syntactic  approach.  In  the 
decision-theoretic  approach,  a  set  of  characteristic  measurements, 
called  features,  are  extracted  from  the  patterns;  the  recognition  of 
each  pattern  is  usually  made  by  partitioning  the  feature  space  L 1 J .  Most 
of  the  developments  in  pattern  recognition  research  during  the  past 
fifteen  years  deal  with  the  decision-theoretic  approach  ana  its 
applications  £1-6].  In  some  pattern  recognition  problems,  the  structural 
information  which  describes  each  pattern  is  important,  and  the 
recognition  process  includes  not  only  the  capability  of  assigning  the 
pattern  to  a  particular  class,  but  also  the  capacity  to  describe  aspects 
of  the  pattern  that  make  it  ineligible  for  assignment  to  another  class. 
A  typical  example  of  this  class  of  recognition  problem  is  picture 
recognition.  In  this  class  of  recognition  problem,  the  patterns  under 
consideration  are  usually  quite  complex  and  the  number  of  features 
required  is  often  very  large,  which  makes  the  idea  of  describing  a 
complex  pattern  in  terms  of  a  composition  of  simpler  subpatterns  very 
attractive.  In  order  to  represent  the  hierarchical  structural 
information  of  each  pattern,  that  is,  a  pattern  described  in  terms  of 
simpler  subpatterns  and  each  simpler  subpattern  again  described  in  terms 
of  even  simpler  subpatterns,  etc.,  the  syntactic  approach  has  been 
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proposed  C 7—1 03 -  T hi s  approach  draws  an  analogy  between  the  structure  of 
patterns  and  the  syntax  of  languages.  Patterns  are  specified  as  being 
built  up  out  of  subpatterns  in  various  ways  of  composition,  just  as 
phrases  and  sentences  are  built  up  Oy  cone «/ ten a* ing  words,  ana  woras  are 
built  up  by  concat enat i ng  characters.  The  simplest  subpatterns  art 
callea  pattern  primitives.  The  language  that  provides  the  structural 
description  of  patterns  in  terms  of  a  set  of  pattern  primitives  ana 
their  composition  operations  is  sometimes  called  the  pattern  description 
language.  The  rules  governing  the  composition  of  primitives  into 
patterns  are  usually  specified  by  the  so-called  grammar  of  the  pattern 


descri pt i on 

Language . 

A  syntactic  pattern  recognition 

system 

can  be  considered 

as 

cons  is t i ng 

of  three  major 

parts,  namely. 

preprocess i ng,  pattern 

description 

or  representa t i on. 

and  syntax 

analysis  (Figure  1.1). 

Tne 

f unct i ons 

of  preprocessing 

include 

(i) 

pattern  encoding 

ana 

approximation;  and  (ii)  filtering,  restoration  and  enhancement. 

An  input  pattern  is  first  coded  or  approximated  by  some  convenient 
form  for  further  processing.  Techniques  of  filtering,  restoration  or 
enhancement  are  used  to  clear  the  noise  and  improve  the  quality  of  the 
coded  patterns.  At  the  output  of  the  preprocessor,  presumably,  we  have 
patterns  of  reasonably  "good  quality".  Each  preprocessed  pattern  is  then 
represented  by  a  language-like  structure.  This  pattern  representation 
process  con si sts  of  pattern  segment  at i on  and  primitive  extraction.  In 
order  to  represent  a  pattern  in  terms  of  its  subpatterns,  we  must 
segmentize  the  pattern  and,  in  the  meant ime, i dent i fy  the  primitives  in 
it.  In  other  words,  each  preprocessed  pattern  is  segmentized  into 
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subpatterns  and  pattern  primitives  basea  on  prespecified  syntactic  or 
composition  operations,  ana,  in  turn,  each  subpattern  is  identifiea  with 
a  given  set  of  pattern  primitives.  At  this  point,  each  pattern  is 
representeu  by  a  set  e*  primitives  with  specked  syntactic  operations. 
The  decision  whether  or  not  the  represent  a t i on  is  synt ac t i ca L Ly  correct 
w i l l  be  made  by  the  ‘’syntax  analyzer"  or  "parser".  When  performing  the 
syntax  analysis  or  parsing,  the  analyzer  can  usually  proauce  a  complete 
syntactic  description,  in  terms  of  a  parsing  tree,  of  the  pattern, 
provided  the  latter  is  syntactically  correct. 

For  proper  representation  of  pattern  structures,  different 
languages  has  been  proposed.  String  language  has  been  appliec  in  the 
problem  of  shape  analysis  and  waveform  analysis  (Figure  1.2).  Tree 
language  has  been  applied  in  the  problem  of  texture  analysis, 
fingerprint  recognition  and  scene  analysis  (Figure  1.3).  Recently 
attributed  grammar  is  proposed  and  PEE  <  primit ive-extract ion-embedded  ) 
parser  is  designed  to  increase  classification  performance  LldJ. 

1,2.2  Time-varying  image 

The  various  methods  applied  to  the  analysis  of  time-varying  images 
can  be  classified  into  two  groups.  For  research  in  the  first  group 
(c ross-cor rela t i on  technique  and  image  differencing  technique),  analysis 
operations  are  conducted  at  image  pixel  level  directly,  while  for  the 
second  group,  actual  analysis  starts  after  some  features  (boundary, 
etc.)  have  been  extracted  from  each  individual  image  in  the  sequence. 
These  techniques  are  further  described  as  follows: 

rroc^-ro*"'-** i  r't  ion  technique  Cl  5,1 8,22,41 ,64,65,661] :  Leese  et  al. 
C 1 8 D  give  a  tyoical  example.  They  compare  two  successive  pictures  with 
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the  first  picture  being  divided  into  systematic  sections  (64x64  pixels). 
Then  for  each  section,  some  reasonable  area  of  the  second  picture  is 
searched  for  a  good  match  to  the  original  section.  They  form  &  cross- 
correlation  coefficient  using  the  fast  Fourier  transform  on  the  full 
gray  values  within  the  section.  The  c ross-cor re lat i on  coefficient  is 
computed  for  each  pairing  of  the  original  section  with  a  candidate 
section  in  the  second  picture.  The  candidate  section  which  yields  the 
maximum  coefficient  is  chosen  as  the  match.  Then  a  motion  vector  is 
computed  as  the  distance  and  direction  between  the  center  of  the 
original  section  and  the  center  of  the  match  section.  The  motion  vector 
is  essentially  assigned  to  the  section,  not  to  any  object  within  the 
section. 

Image  differencing  technique  [23,30,31 ,46,47,67,68,69,70 j :  In 
[46,47],  the  image  differencing  technique  is  applied  to  find  the 


variat ion 

between 

two 

images  of  the  same  scene. 

The 

images  are 

carefu  l  ly 

a  l  i gned 

by 

both  spatial  coordinates  and 

gray 

value.  The 

spatial  registration  is  done  by  considering  one  image  as  the  reference 
image  and  then  distorting  the  other  image  until  they  are  aligned.  The 
distortion  is  a  localized  procedure  which  operates  on  subregions  of  the 
images.  Cross-correlation  technique  is  used  to  compute  the  amount  of 
distortion  necessary  to  align  a  subregion  with  its  corresponding 
subregion  in  the  reference  image.  After  the  spatial  registration  has 
been  completed,  a  poi nt-to-poi nt  subtraction  process  generates  a  third 
image  which  displays  the  variation  between  the  given  images. 

Limb  and  Murphy  116/3  report  a  hardware  implementation  of 
subtractive  method.  In  addition  to  the  subtraction  process  between  the 
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corresponding  pixels  of  two  consecutive  images,  wi thin-image  comparisons 
are  made  among  pixels  and  their  suitable  neighbors.  The  withm-image 
comparisons  are  used  to  normalize  the  between-i mage  comparisons.  These 
comparisons  are  essentially  the  absolute  differences  of  the  pixel  gray 
values  and  are  summed  over  the  entire  image.  This  yields  a  velocity 
estimate  for  the  image  as  a  whole  and  not  for  any  specific  object. 

The  research  work  in  group  II  includes 
[37,39,42,43,48,49,72,73,74,773 :  Chow  and  Aggarwal  l433  develop  an 
algorithm  to  track  moving  objects  in  an  image  sequence.  After 
preprocessing  of  an  image,  they  build  a  model  which  consists  of  the 
objects  extracted  from  the  image.  Each  object  is  represented  as  a  set 
of  feature  values;  area,  centroid  position,  etc.  The  matching  process 
of  moving  objects  between  consecutive  images  is  performed  through  the 
comparison  of  these  feature  values.  After  the  matching,  the  model  is 
then  updated  with  new  information.  The  analyzed  results  include  the 
number  of  moving  objects  and  their  corresponding  velocities  in  the  whole 
sequence.  Other  similar  works  include  [37,39,42,49,773. 

Recently  O'Rourke  and  Badler  report  a  system  for  the  analysis  of 
human  motion  images  [733.  The  model  for  human  body  is  composed  of 
segments  and  joints.  A  joint  is  a  unique  point  connecting  two  segments. 
A  segment  is  an  abstract  rigid  body  with  an  associated  embedded 
coordinate  system.  The  surface  of  each  segment  is  defined  by  a 
collection  of  graphical  primitives,  called  spheres,  located  at  fixed 
positions  within  the  segment's  coordinate  system.  There  are  constraints 
governing  the  relations  between  different  parts  of  the  body.  These 
constraints  arise  from  the  structure  of  the  body,  gravity  effect,  etc. 
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Based  on  the  analysis  of  the  past  images,  the  system  forms  a  motion 
description  which  is  comprised  of  a  set  of  piecewise  linear  functions. 
Each  linear  function  describes  the  location-time  relation  of  a  specific 
feature  point.  The  model  is  updated  with  new  information.  The  system 
then  predicts  searching  regions  of  the  parts  for  the  analysis  of  next 
image.  If  the  prediction  fails  for  a  specific  part,  then  the  system 
terminates  the  analysis  process  for  this  part  and  works  for  others 
hoping  that  the  information  from  succeeding  images  will  resolve  the 
confusion.  The  model  presented  here  is  essentially  an  attributed 
grammar,  but  with  a  slightly  different  form.  Since  it  is  focused  on  a 
specific  object — human  body,  composed  of  24  segments  and  25  joints,  the 
dynamic  process  of  the  image  sequence  is  represented  as  a  structure- 
preserved  variation  of  attributed  language.  In  other  words,  each  image 
is  represented  by  a  set  of  feature  values  and  the  whole  sequence  is 
represented  by  a  set  of  piecewise  linear  equations.  Besides,  it  is  not 
known  whether  or  not  the  relaxation  process  of  uncertain  part  will 
eventually  be  stable,  which  is  the  common  problem  in  Yachida  et  at.  1772 
and  Tsotsos  et  al.  L7h3. 

Tsotsos  et  al.  C74]  report  a  system  for  the  analysis  of  the  image 
sequence  of  left  ventricular  wall  motion.  They  propose  a  way  to 
represent  motion  concepts  based  on  the  semantic  network  theory.  Each 
motion  concept  is  associated  with  and  defined  by  a  "frame."  A  frame  is 
definable  by  the  user.  Frames  have  an  arbitrary  number  of  "slots"  that 
form  their  parts.  Slots  come  in  two  varieties:  "prerequisite"  and 
"dependent."  Prerequisite  slots  specify  concepts  that  must  be  observed 
before  the  frame  can  be  instantiated,  while  dependents  provide 
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additional  semantic  components  that  are  included  along  with  the  frame 
concept  on  instant iation.  For  example,  the  concept  of  "area  change"  ana 
"contract"  is  defined  as  follows: 

frame  area_change  with 
prerequi si tes 

subj :  contract i  le__object; 
time_int :  t ime_inte rva l ; 
start_a:  area_value; 
end_a:  area_va lue; 

end 

frame  contract  is_a  area_change  with 
prerequi si tes 

start__a:  such  that 

start__a  >  end__a; 

dependents 

speed:  speed__v  with 

speed  +  (start_a-end_a)  t  time__int.  duration 

end 

These  two  frames  illustrate  most  of  the  "syntact ical"  construct  of 
their  representation  formalism.  From  the  viewpoint  of  Fu  L 1 0 J  and  Tsai 
and  Fu  [143,  the  representation  method  of  motion  information  described 
above  is  essentially  that:  each  motion  concept  is  represented  as  a  set 
of  prespecified  attributes,  each  of  which  has  a  special  meaning. 
Tsotsos  et  al.  analyze  the  image  sequence  using  hypotheses-cooperat ion 
method:  uncertain  parts  in  an  image  are  updated  using  a  relaxation 
process  [883,  during  which  the  iteration  is  not  completed  until  a 
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convergence  to  stable  condition  is  achieved.  It  is  not  sure  whether  or 
not  the  iteration  process  will  always  lead  to  convergence  in  every 
uncertainty  case. 

Yachida,  Ikeda  and  Tsuji  C76J  analyze  the  behavior  of  heart  wall 
motion  by  measuring  the  thickness  of  heart  wall  in  each  image.  The 
thickness  of  the  heart  wall  is  measured  at  a  set  of  boundary  points. 
The  thickness  is  determined  as  the  distance  between  the  points  of  the 
internal  boundary  and  the  points  of  the  external  boundary  intersecting 
with  a  set  of  lines  perpendicular  to  the  internal  boundary.  After  the 
feature  extraction  of  the  whole  image  sequence,  the  dynamic  process  of 
the  heart  wall  motion  is  represented  as  a  3-D  surface  map,  with  x-axis 
being  the  spatial  domain,  y-axis  being  the  time  domain  and  z-axis  being 
the  thickness.  The  motion  of  heart  wall  is  analyzed  by  observing  the 
surface  configuration.  Heintzen  et  al.  C933  also  represent  the 
information  extracted  from  the  heart  motion  images  as  a  3-D  surface, 
while  Garrison  et  al.  C90, 91 ,92,94,95 j  use  a  curve  (volume-time)  to 
represent  the  dynamic  process. 

The  various  research  areas  of  time-varying  images  are  briefly 
described  as  follows: 

Aerial  and  satellite  images:  Evaluation  of  aerial  and  satellite 
image  sequences  is  an  active  area  in  the  problem  of  time-varying  image 
analysis  [15,18,20].  Wind  velocities  are  estimated  from  cloud 
displacements  observed  in  a  sequence  of  satellite  images  as  regular 
input  data  to  weather  forecasting.  Hubert  et  al.  121]  combine  two  or 
more  copies  from  a  sequence  of  satellite  images  taken  at  2U-3U  minute 
intervals  over  a  period  of  2-3  hours  into  a  film  loop  which  is 
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continuously  projected  onto  a  digitizing  tablet.  An  operator  selects  a 
cloud  feature  (e.g.  cyclonic  vortices,  cold  fronts)  and  marks  us 
position  in  the  projection  of  initial  and  final  frame  from  the  film  loop 
,  using  the  intermediate  frames  to  securely  track  the  selected  featjres. 
The  measured  displacement  of  the  selected  feature  is  corrected  for 
distortion  and  transformed  to  an  earth  surface  coordinate  system.  On  tne 
hypothesis  that  positional  changes  of  clouds  are  solely  affecteo  Dy 
horizontal  winds,  the  resulting  cloud  feature  displacement  is  converted 
into  wind  velocity. 

Traffic  monitoring:  Several  investigations  have  been  reported  to 
detect  moving  vehicles  in  sequences  of  film  or  video-films  of  traffic 
scenes  and  to  track  their  motion.  A  reliable  solution  to  this  problem 
would  not  only  allow  to  count  vehicles,  but  in  addition  enable  the 
ident if ication  of  vehicle  type  (car,  truck, bus  ,...)  and  the  observation 
of  vehicle  behavior  for  a  variety  of  quickly  changing  traffic 
situations.  Wolferts  £223  described  an  interactive  setup  to  measure 
vehicle  velocity  on  time-lapsed  film  using  cross-corre lat ion  to  track 
their  images.  Onoe  et  al.  £233  evaluated  video  sequences  for  this 
purpose.  Jones  £243  reported  on  real-time  tracking  of  features  on 
vehicles  in  video-sequences.  Jain  et  al.  £25]  used  video-sequences  of 
traffic  scenes  to  study  the  detection,  isolation,  tracking,  and 
description  of  moving  objects. 

Industrial  automation:  Application  of  visual  sensors  to  industrial 
automation  appears  to  be  another  area  of  time-varying  image  problems 
£263.  Newmann  £273  studies  the  tracking  of  an  object  on  a  simulatea 
conveyor  belt  by  instantiating  2-D  relational  models  on  straight  line 
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contour  approx imat ions  to  images  of  a  scene  that  contained  a  moving 
object.  If  tracking  has  to  be  performed  under  real-time  constraints,  one 
reverts  to  registering  greyvalue  or  binary  templates  or  to  the 
identification  of  a  feature  that  is  cha rac^r i st i c  for  the  object  in 
question  and  can  be  easily  isolated  within  a  search  window.  Uno  L28J 
employed  such  techniques  on  using  a  TV-camera  to  detect  the  position  o4 
bolts  and  reinforcement  ribs  on  moving  steel  moulds  in  order  to  control 
a  manipulator.  Jones  et  al.  L 24]  investigated  real-time  tracking 
techniques  to  study  the  application  of  fast  visual  feedback  to 
industrial  automation.  Eskenazi  C29]  investigated  a  real-time  tracking 
capability  based  on  greyvalue  correlation  by  using  the  video  sensing  ana 
processing  setup  developed  for  the  navigable  robot,  by  simu Itaneous ly 
tracking  the  same  object  in  two  stereo  image  sequences,  it  is  able  to 
determine  the  object's  3-D  trajectory.  Using  this  technique  to  track  a 
stationary  object  from  a  moving  robot,  the  robot's  trajectory  can  be 
determined  in  order  to  guide  its  navigation  and  to  adjust  the  pan  ana 
tilt  of  a  stereo  camera  assembly  on  the  robot  for  keeping  the  reference 
object  in  the  field  of  view. 

Medical  applications :  Medicine  provides  a  major  area  for  image 
sequence  analysis.  The  subtraction  of  X-ray  images  obtained  before  and 
after  injection  of  roentgen-opaque  material  into  the  blood  enhances  the 
resulting  difference  image  for  interpretation  by  a  physician. 
Digitization  of  X-ray  film  images  facilitates  the  performance  of 
nonlinear  operations  such  as  compensation  for  film  charac teri st i cs 
during  the  de terminat ion  ot  the  difference  image.  Although  digitized 
film  images  have  to  be  aligned  prior  to  subtraction,  relative  geometric 
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distortions  can  usually  be  neglected.  Therefore,  the  registration  tends 
to  be  much  less  involved  here  than  with  satellite  images.  If  an  average 
of  several  digitized  pre-injection  images  is  subtracted  from  a  post¬ 
injection  image,  image  components  not  affected  by  the  injection  of 
roentgen-opaque  material  can  be  suppressed  even  more  effectively 
C30,313. 

Sequences  of  X-ray  images  from  the  left  ventricle  have  been 
evaluated  interact i vely  by  cardiologists  to  determine  the  Left  ventricle 
volume  or  to  search  for  abnormali t ies  in  ventricle  wall  motion.  One  of 
the  main  problems  consists  in  a  reliable  determination  of  the  ventricle 
outline  in  the  X-ray  image.  Chow  and  kaneko  L3Q3  search  a  suitably 
enhanced  digitized  X-ray  film  image  of  the  left  ventricle  for  subregions 
with  Larger  greyvalue  variances.  For  such  subregions,  a  mixture  of  two 
normal  distributions  is  fitted  to  the  corresponding  greyvalue  histogram. 
If  an  acceptable  fit  is  obtained  for  a  subregion  a  threshold  is 
determined  which  is  used  to  classify  the  greyvalues  of  this  subregion  as 
either  interior  or  exterior  to  the  left  ventricle.  In  this  way,  a  series 
of  25  left  ventricular  contours  is  derived  automatically  covering 
approximately  one  cardiac  cycle. 

Injection  of  radioactive  nuclei  into  a  peripheral  vein  and 
subsequent  recording  of  their  decay  gamma  rays  by  a  sc i ent igraphi c 
camera  provides  another  way  to  obtain  images  of  the  left  ventricle. 
Although  these  scientigraphic  images  offer  even  less  contrast  than  gooa 
X-ray  images,  Hachimura  et  al.  L 32 3  were  able  to  determine  the  left 
ventricular  contour  in  such  sc ient igraphi c  image  sequences,  by  aetailea 
analysis  of  the  left  ventricular  contour  obtained  at  the  end-diastole 


and  the  end-systole  they  attempt  to  classify  the  observed  left 


ventricule  wall  motion  into  a  normal  and  an  abnormal  category.  Based  on 
a  time  series  of  nine  scient igraphi c  images  observed  over  1UU  msec 
intervals  after  the  end-diastole  they  the  left  ventricular 
volume  and  plot  it  versus  time. 

A  3-D  distribution  of  greyvalues  reflecting  the  spatial 
conf iguration  of  organs  in  a  Living  body  can  be  determined  oy 
computerized  tomograpny.  A  three  layer  approx imat ion  to  such  a 
description  tor  the  thorax  of  a  Living  dog  is  given  Dy  Johnson  et  al. 
C 33 3 .  If  position  ,size  and  shape  of  an  organ  varies  with  time  as  in  tne 
case  of  the  left  ventricle,  such  a  variation  can  be  represented  by  a 
time  series  of  3-D  greyvalue  di s t r i but i ons  which  can  be  considered  as  a 
four-dimensional  greyvalue  distribution.  For  this  purpose,  Herman  ana 
Liu  C  34  3  generalized  a  search  algorithm  by  Liu  L35j  for  the 
determination  of  a  2-D  surface  in  a  3-D  greyvalue  distribution  such  as 
obtained  by  3-D  computerized  tomography. 

Information  about  dynamic  changes  in  size,  shape  and  position  on 
intact  working  organs  is  of  great  interest  not  only  for  the  detection  of 
abnormalities  but  even  for  a  detailed  understanding  of  their  function. 
This  is  not  restricted  to  time  series  of  3-D  greyvalue  distributions  as 
obtained  from  computerized  tomography.  Heintzen  et  al.  L3bJ  emphasize 
this  strong  connection  between  time  sequences  of  single-plane  X-ray 
images  and  a  better  understanding  of  certain  organ  functions. 

Beyond  pure  medical  app  l  i cat  ions ,  the  evaluation  of  image  sequences 
spreads  into  biophysics  and  biology.  For  example,  Yachioa  et  al.  L 3 7 j 
observe  fishes  swimming  in  a  vat  by  an  overhead  TV-camera  connected  to  a 
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video-tape  recorder  to  study  their  behavior  unaer  a  variety  of  stimuli 
such  as  lights  or  tones.  About  8  frames  per  second  are  recordea  for 
periods  between  2  and  30  seconds,  resulting  in  sequences  of  20-250 
frames.  Since  the  images  of  these  moving  objects  are  usually  blurred, 
temporal  as  well  as  spatial  greyvalue  differences  are  used  to  separate 
the  images  of  moving  objects  from  those  of  stationary  scene  components. 
Results  from  a  previous  frame  are  usea  to  guide  the  feature  extraction 
process  in  the  subsequent  frame.  They  model  the  essential  parts  of  the 
scene  as  it  is  presented  in  each  frame  and  exploit  such  models  obtained 
from  neighboring  -  prior  as  well  as  posterior  -  frames  to  deduce 
uncertain  parts  or  reanalyze  them.  Davenport  et  al.  L 383  also  study  the 
stimulus-response  behavior  of  mi croorgani sms  whose  movement  in  a  wet- 
slide  preparation  under  a  microscope  is  recorded  on  video  tape. 

Ariki  et  al.  L39T  design  an  elaborate  interactive  facility  for  the 
analysis  of  image  sequences.  They  analyze  the  morphogenetic  movement  of 
a  dissociated  cell  of  Xenopus  laevis,  a  protozoon.  Operator  intervention 
is  required  to  check  frame  registration  (low  level  interaction).  Meaium 
level  interaction  is  employed  to  define  a  model  for  the  object  that  has 
to  be  traced  throughout  the  image  sequence.  Such  medium  level 
interaction  enable  the  evaluation  of  a  variety  of  real  world  scenes. 
Takagi  et  al.  C403  conduct  a  tracking  problem  in  which  to  determine  the 
paths  of  gramules  in  a  cultivated,  living  pancreatic  cell  recoroea  by 
time-lapse  cinemi crophotography  at  1  second  intervals  for  5-10  minutes. 
These  granules  carry  insulin  from  ribosomes  where  it  is  produced  to  the 
cell  membrane.  Analyzing  the  motion  of  these  granules  may  tnerefore 
contribute  to  a  better  understanding  of  hormane  production  mechanism. 
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One  cf  the  problems  in  time-varying  image  analysis  is  the  oroper 
representat ion  of  information  extracted  from  the  image  sequence.  In 
syntactic  pattern  recognition,  a  pattern  is  usually  representeo  by  a 
linguistic  notion  called  a  sentence.  The  serf3'  ~e  could  be  a  string  ,  a 
tree  or  a  graph  of  pattern  primitives  and  relations.  Most  of  the 
developments  in  syntactic  pattern  recognition  research  during  the  past 
two  decades  deal  with  'static  patterns'  te.g.  fingerprint,  character, 
chromosome  shape).  For  static  patterns,  given  an  input  pattern  x  ana  a 
set  of  pattern  grammars  representing  different  classes,  parsing  scheme 
is  applied  for  the  classification  of  input  pattern.  While  for  time- 
varying  patterns,  after  finding  a  proper  representat ion,  we  need  to  not 
only  analyze  the  pattern  at  each  single  stage,  but  aiso  test  the 
mechanism  charact eri z ing  the  dynamic  process  of  the  sequence. 

1 .3  Summary  of  the  Contents 

Chapter  2  is  concerned  with  the  time-varying  pattern  analysis  using 
a  syntactic  method.  (For  convenience,  we  use  TV  as  an  abbreviation  of 
t i me-vary i ng .  In  formal  language  theory,  a  translation  is  defined  as  a 
mapping  from  a  language  to  another  language  L^.  If  we  consider  as 
the  set  of  possible  patterns  occurring  at  time  t^,  as  the  set  of 
patterns  at  time  t^,  then  the  relation  governing  the  TV  phenomena  could 
oe  formulated  as  a  translation  problem.  In  chapter  2,  we  first  explore 
such  a  translation  using  a  pattern  deformation  model  Li  1 J  and  then 
consider  the  appli cabi l i ty  of  the  translation  models  to  time-varying 
pattern  analysis.  Furthermore,  in  terms  of  the  translation  models,  we 
will  formulate  the  problem  of  TV  pattern  analysis  as  one  which  can  be 
solved  using  traditional  method,  specifically,  context-free  programmed 
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Language  parsing. 

In  chapter  3,  the  concept  of  language  translation  is  extended  from 
strings  to  trees  using  a  generalized  syntax-di rect ea  model. 
Convent ionat  top-down  and  bottom-up  tree  translation  moaels  are  compared 
with  the  generalized  model.  It  is  shown  tnat  both  top-down  and  bottom- 
up  models  are  special  cases  of  the  generalized  model.  A  parsing 
algorithm  for  this  generalized  tree  translation  model  is  also  presented. 
A  traffic  monitoring  experiment  is  described  in  chapter  4.  Each  image 
of  the  traffic  scene  is  divided  into  a  set  of  windows  which  is  then 
represented  as  a  tree  structure.  A  tree  translation  schema  is  applieo 
to  describe  the  motion  of  vehicles  in  the  input  image  sequence,  the 
matching  process  of  vehicles  between  consecutive  images  is  performed 
through  a  tree  translation  parsing. 

Chapter  5  summarizes  the  results  of  this  study  and  proposes 
suggestions  for  further  research. 
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CHAPTER  2 

SYNTAX  OF  TIME-VARYING  PATTERNS 

2_ Int  roduct i  on 

In  syntactic  pattern  recognition,  a  pattern  is  usually  representea 
by  a  linguistic  notion  called  a  sentence  C10J.  The  sentence  coula  De  a 
string,  a  tree,  or  a  graph  of  pattern  primitives  and  relations.  %‘ost  ot 
the  developments  in  syntactic  pattern  recognition  research  during  the 
past  two  aecades  deal  with  "static  patterns"  Ce.g.  chromosome 
c  lass i f i cat  ion,  character  recognition,  and  fingerprint  classification). 
For  static  patterns,  given  an  input  pattern  x  and  a  set  of  pattern 
grammars  representing  different  classes,  error-correct i ng  parsers  are 
used  to  classify  x  into  one  c-f  the  classes  L.10,113.  While  for  TV 
patterns,  given  an  input  seauence  x^,  x^,  ...,  we  need  to  not  only 
analyze  the  pattern  at  each  single  stage,  but  also  test  the  mechanism 
characterizing  the  dynamic  process  of  the  sequence. 

In  formal  language  theory,  a  translation  is  defined  as  a  mapping 
from  a  language  to  another  language  L^.  If  we  consider  as  the  set 
of  possible  patterns  occurring  at  time  t^,  as  the  set  of  patterns  at 
time  t^,  then  the  relation  governing  the  TV  phenomena  could  be 
formulated  as  a  translation  problem.  In  this  chapter,  we  first  explore 
such  a  translation  using  a  pattern  deformation  model  L 1 1 J  and  then 
consider  the  app li cabi l i ty  of  the  translation  models  to  time-varying 
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pattern  analysis.  Furthermore,  in  terms  of  the  translation  moaels,  we 
will  formulate  the  problem  of  TV  pattern  analysis  as  one  which  can  oe 
solved  using  traditional  method,  specifically,  context-free  programmed 
language  parsing, 

_2.^2  Preliminaries 

Definitions  ana  notations  that  will  be  referred  to  in  tnis  chapter 
are  summarized  as  follows  C51-54D. 

Def ini tion  2.1  :  Suppose  that  £  is  an  input  alphabet  ana  A  is  an  output 
alphabet.  We  define  a  translation  from  a  language  C  £  to  a 
language  !r  A  as  a  relation  T  from  2^  to  A  such  that  the  domain  of 
T  is  and  the  range  of  T  is  - 

Definition  2.2:  A  syntax-directed  translation  schema  (SDTS  for  short)  is 
a  5-tuple  T  =  (N,£,  A,  R,  S),  where 


CD  N  is  a  finite  set  of  nonterminal  symbols, 

(2)  £  is  a  finite  input  alphabet, 

(3)  A  is  a  finite  output  alphabet. 


(4) 

R  is  a 

finite 

set  of 

rules 

of 

the  form  A  ♦  a. 

6,  where 

a  e  (N 

uD% 

3  e  (N 

U  A)*, 

and 

the 

nonterminals  in 

3  are  a 

permutation  of  the  nonterminals  in  a, 

(5)  S  is  a  distinguished  nonterminal  in  N,  the  start  symbol. 

Let  A  +  a,  B  be  a  rule.  To  each  nonterminal  of  a  there  is  an  associated 
identical  nonterminal  of  6.  If  a  nonterminal  B  appears  only  once  in  a 
and  6,  then  the  association  is  obvious.  If  B  appears  more  than  once,  we 
use  integer  superscripts  to  indicate  the  association.  This  association 
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is  an  intimate 

part  of 

the  rule.  For  example. 

in  the 

ru  le 

,  ^  ( 1 )  (2) 

A  ^  B  CB  , 

b(2)b(1)c. 

the  three  positions  in 

. (1 ) r t(2) 
b  CB 

are 

associated  with 

positions  2, 

3,  and  1,  respect i ve ly,  in 

b(2)b(1)c. 

The 

translation  defined  by  T,  oenoted  t(T),  is  the  set  of  pairs 

★ 

{(x,y)|<S,S)  ===>  (x,y),  x  e  £  and  y  e  A  >. 

Definition  2.3:  If  T  =  (N,]Cr  A,  S)  is  an  SDTS,  then  t(T)  is  called  a 
syntax-directed  translation  (SDT).  The  grammar  G_.  =  (N,  £,  ps  S),  where 

P  =  {A  +  a  |  A  +  a,  6  is  in  R>, 

is  called  the  input  grammar  of  the  SDTS  T.  The  grammar 

i  i 

G  =  (N,  A,  P  ,  S>,  where  P  =  {A  S  |A  *►  a,  8  is  in  R>  is  called  the 
output  grammar  of  T. 

Definition  <2.4:  An  SDTS  T  =  (N,£,  A f  R,  S)  such  that  in  each  rule 
A  -►  a,  8  in  R/  associated  nonterminals  occur  in  the  same  order  in  a  and 
6  is  called  a  simple  SDTS.  The  translation  defined  by  a  simple  SDTS  is 
called  a  simple  SDT. 

Definition  2.5:  A  simple  SDTS  T  =  (N,  A,  R,  S)  such  that  each  rule 
in  R  is  either  of  the  form  A  •*>  a B,  aB  or  of  the  form  A  +  a, a  where 
^  B  e  N,  a  e  and  a  e  A*,  is  called  a  regular  SDTS.  The  translation 
defined  by  a  regular  SDTS  is  called  a  regular  SDT. 
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Definition  2.6:  For  two  strings,  x,  y  c  £  ,  we  define  a  transformation 
T:  -►  such  that  y  c  T(x).  The  following  three  transformations 

are  introduced  C54J: 

(1)  substitution  error  t ransformat ion 

Ts 

to ^ a |  _ —  to  ^  b  to  ^  ,  for  all  s /  b  c  5 ,  a  ^  b , 


(2)  deletion  error  transformation 


| - ^^2'  f°r  a  e  S  * 

(3)  insertion  error  transformation 


U)1a)2  ^ - w1au>2,  for  all  a  e  £  where  ^  e  53 


Example  2,1 ;  Given  a  sentence  x  =  cbabdbb  and  a  sentence  y  =  cbbabbab, 
then 

Ts  Ts  TI 

x  =  cbabdbb  | -  cbabbbb  | -  cbabbdb  | -  cbbabbdb  =  y 


Definition  2,7:  A  context-free  programmed  grammar  (CFPG)  is  a  5-tuple  G 
=  (N,  ]£/  J,  P,  S)  where 

(1)  N  is  a  finite  set  of  non-terminals, 

(2)  53  is  a  finite  set  of  terminals. 


(3)  S  is  the  start  symbol  in  N, 

(4)  P  is  a  finite  set  of  programmed  productions 
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(5)  J  is  a  finite  set  of  production  Labels. 

Each  production  in  P  consists  of  a  Label  reJ,  a  core  production  of 
the  torn  A  -*■  a  where  AeN,  a  c  (N  U  ]£)*,  and  a  success  branch 
fielo  and  3  failure  branch  field  each  consisting  of  elements  from 

J.  (This  definition  allows  core  production  to  be  A  ♦  A,  whe^e 

AeN  and  A  is  an  empty  string.) 

A  derivation  or  generation  in  6  proceeds  as  follows:  The  first 

production  is  applied  to  the  start  symbol  S;  therefore,  if  production  r 
is  applied  to  the  current  sentential  form  y  to  rewrite  a  nonterminal  A, 
and  if  y  contains  at  least  one  occurrence  of  A,  then  the  leftmost  A  is 
rewritten  by  the  core  of  production  r  and  the  next  production  label  is 
selected  from  the  success  branch  field  of  r;  if  the  current  sentential 
form  does  not  contain  A,  then  the  core  of  production  r  cannot  be  used 

and  the  next  production  label  is  selected  from  the  failure  branch  field 

of  r;  if  the  applicable  branch  field  is  empty,  the  derivation  halts. 

2.3  Formulat ion  of  TV  Patterns  as  _a  CFPL 
To  recognize  noisy  syntactic  patterns,  various  types  of  error 
t ransformat i on  models  were  proposed.  Fung  and  Fu  L52D  proposed  a 
st ructure-preserved  deformation  model  to  handle  substitution  error  for 
string  patterns.  Lu  and  Fu  Li 1 D  extended  the  model  to  include 
substitution  error,  insertion  error  and  deletion  error,  which  were 
introduced  in  the  previous  section.  It  is  known  that  any  pattern  x  can 
be  transformed  into  any  other  pattern  y  by  a  sequence  of  error 
transformations.  If  we  analyze  a  TV  pattern  x^,  x^,  ...  x^  from  the 
viewpoint  of  deformation  model,  then  the  TV  phenomena  can  well  be 


24 


interpreted  through  a  sequence  of  property  selected  error 

t ransformat ions  which  transforms  into  x^,  x^  into  *  y  ...,  and  x^_^ 

into  x  . 
n 

The  deformation  models  proposed  by  Fung  and  Fu,  and  Lu  and  Fu  can 
be  formulated  in  terms  of  translation  schema.  Consider  patterns  which 
are  described  by  a  regular  grammar.  To  handle  substitution  error,  a 
regular  SDTS  T(N,  £,  A,  R,  S)  can  be  constructed,  where  £  =  A  is  the 
primitive  set,  R  is  a  set  of  rules  of  the  form 

A  +  aB,  bB  A,  B  e  N,  a,  b  e  £ 

For  each  rule  A  ♦  aB  in  the  regular  grammar  and  each  terminal  b  e  £, 
add  a  rule  A  +  aB,  bB  to  R.  To  include  insertion  error  and  deletion 
error,  for  each  rule  A  -►  aB  in  the  regular  grammar,  add  a  rule  A  +  aB, 
aB  to  R  for  each  a  in  uz2  •••  u£K  lKe>),  where  K  is  a  parameter 

and  e  is  the  empty  string.  For  patterns  which  are  describee  by  a 
context-free  grammar,  a  simple  SDTS  can  be  constructed  to  model  the 
three  types  of  deformation.  First  of  all,  the  grammar  is  transformed 
into  its  Greibach  Normal  Form  C513.  Then  the  construction  procedure  of 
a  simple  SDTS  is  similar  to  that  of  a  regular  translation  schema  except 
that:  For  each  rule  A  ■*  aB  in  the  context-free  grammar,  add  A  aB,  aB 

for  each  a  e(Z^  U  ***  u  U  te»  - 

As  far  as  error  modeling  is  concerned,  Lu  and  Fu's  model  can  handle 
any  number  of  errors  occurred.  For  time-varying  patterns,  we  restrict 
the  number  of  errors  no  larger  than  K.  The  difference  between  Lu  and 
Fu's  model  and  this  translation  model  is  that  Lu  and  Fu's  model  handle 
errors  implicitly  (or  recursively)  while  this  translation  model  must 


m 
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explicitly  specify  the  kind  of  errors  it  intends  to  cover.  For  example, 
in  Lu  and  Fu’s  model  E  a  and  E  ♦  bE  recursively  describe  any  number 
of  errors  of  inserting  b  in  front  of  a.  But  in  this  t.  dnslation  model, 
a  translation  rule  A  +  aB,  ctB  has  to  be  included  to  cover  each 
individual  error  a  -►  a  even  though  a  has  no  limitation.  The  parameter  k 
here  only  indicates  that  the  number  of  errors  a  ♦  a  is  finite.  In  other 
words,  k  could  be  any  positive  integer  but  k  <  Essentially  we  use 

the  translation  schema  to  implement  the  deformation  model.  To  make  it 
suitable  for  an  n-stage  TV  pattern  analysis  problem  with  sequence  % 
x2/  — '  xn+i/  we  the  following  definition. 

Definition  2.8:  An  n-stage  SDTS  is  a  5-tuple 

T  =  (N,  A,  R,  S),  where 

(1)  A,  S  are  defined  as  those  of  an  SDTS. 

(2)  R  is  a  set  of  rules  of  the  form  A  +  a,  ,  B^,  ...,  S  .  where 
a  e  (N  U  5^)*,  6..  e  (N  U  A)*,  and  the  nonterminals  in  6^  are  a 
permutation  of  the  nonterminals  in  a. 

The  translation  defined  by  an  n-stage  SDTS  T,  denoted  by  x(T),  is  the 
set 

★ 

yv  •**  yn)  I  (S,  S,  s)  ===>(x,  yv  yn),  X  e  A*> 

Thereafter  the  analysis  of  TV  patterns  becomes  a  translation  problem. 

Next  we  investigate  the  "parsing"  of  a  translation  beginning  with  a 


definition: 


26 


Definition  2.9:  Let  T  be  a  one-stage  SDTS 

* 

t (T)  =  L(x,  y)  |  (S,  S)  ===>  Cx,  y) > . 

Define  the  input  language 

L^  as  Lx  |  (x,  y)  e  t(T)  for  all  y> 
and  the  output  language 

as  Ly  |  (x,y)  e  t(T)  for  all  x>- 

The  concatenation  of  and  L0  with  respect  to  T  is  defined  as  the 
set  Lxy  )  (x,  y)  e  t(T)>,  denoted  by  L^. 

L12  -  where  =  Lxy  |  x  e  ,  y  e  L^>. 

Theorem  2,1 :  Let  and  be  the  input  language  and  output  language 
respectively  of  a  regular  SDTS  T(N,  23,  A,  R,  S)  then  l  is  a  CFPL. 
(L>j  is  a  context-free  language,) 

£roof:  We  shall  prove  the  theorem  by  constructing  a  CFP6 

6  such  that  1(6)  =  L^. 

«  • 

A  CFPG  G(N  ,  23  ,  J,  P/  S)  i s  constructed  as  follows: 

Step  1:  N*  =  -CA1  |  A  e  N>  U  (A2  |  A  e  N)  U  <S>. 

Step  2:  23  =  23  U  A. 

Step  3:  Since  T  is  a  regular  SDTS,  each  production  in  R  is  of  the  form: 

where  A,  B  e  N,  a  e  23 , 


A  *►  aB,  aB  or  A  -►  a,  a 


a  c 


* 

A  . 
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(1)  Add  S  +  S]S^  ^CK  |  K  is  the  Label  of  production  beginning  with 

S7>  <t>  to  P. 

(2)  If  A  ♦  a B,  aB  is  a  production  in  R  such  that  A,  B  e  N,  a  c  £ 

and  a  e  A*,  then  add  the  production  to  P: 

2  2  1 
A  aB  <CK|K  is  the  label  of  production  beginning  with  B  >  $ 

(3)  For  each  input  rule  of  R:  A  aB  where  A,  B  e  N  and  a  e  21, 

1  1 

add  the  production  to  P:  A  -►  aB  S(u)  4>,  where  S(u)  is  the  set  of 
labels  of  the  productions  added  in  (2)  corresponding  to  the  translation 
rules  with  input  rule  A  +  aB. 

(4)  If  A  +  a,  a  is  a  production  in  R,  where  A  c  N,  a  e  £  and 

*  2 

a  e  A  ,  then  add  the  production  A  ♦  a  $  $  to  P.  , 

(5)  For  each  input  rule  of  R:  A  ♦  a  where  A  z  N  and  a  e  add 

i 

the  production  to  P:  A  -►a  S(u)  $  where  S(u)  is  the  set  of  Labels  of 
the  productions  added  in  (4)  corresponding  to  the  translation  rules  with 
input  rule  A  -►  a. 

Then  we  shall  prove  that  =  L(G). 

Let  xy  be  in  L^.  Then  by  Definition  2.9  there  exists  a  derivation 
sequence 

(S,  S)  ==>  Ca^  - "=>  - an'  a1a2 - an^  = 

,  a.  c  A  and  A.  e  N.  From  the  construction  procedure  of 
CFPG  G,  it  is  known  that:  For  each  derivation  step 


(a1a2  ai Ai ' 


°1a2" 


‘a-A-)  ==>  (a^a^- 


"ai +1Ai +1 ' 


a1a2" 


*ai+1Ai+1)' 


there  exist  the  productions 
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.1  A1 

Ai  "  a1+1  Ai  +  1 


dna  Ai  "  a1+1  Ai+1  in  P 


with  the  success  field  of  the  first  production  containing  the  label  of 

the  second  production.  8esides  the  success  field  of  the  production 

2  2  11 
A  +  a.  *  A  „  contains  the  laoel  of  the  production  A.  -  +  a.  -  A.  9  ^or 

ii+li+l  i+ii+c:iT^ 

1  <  i  <  n.  Therefore,  there  exists  a  derivation  in  G: 


S  -->  s^2  -=>  a-jA-jS  — >  a1A1a1A1 


=“>  a'jd2A2a1A1  > 


a1 a2A2a1 a2A2 


==>  a1a2 


a  a.a.  *  * #  oc  1  A  1 
n  12  n-1  n-1 


==>  a^a2  •••  anaia2  *“  an-1an  =  xy*  Thus'  Lli;~  L(6)-  Similarly,  from 
the  definition  of  G,  it  can  be  shown  that  L(G) C  i.  Therefore, 


L12  =  L(G). 


Definition  2.10:  Let  T  be  an  n-stage  SDTS,  L1  be  its  input  language,  L^, 
L-5  •••  Ln+1  be  its  1st,  2nd,  nth  output  languages  respectively. 
Define  the  concatenation  of  L^,  L2,  ***  *-n+i  with  respect  to  T  as 

<V2  *•*  xn+1  1  (x1'x2'  ***'  W  e  T(T)>  denoted  as  Li23**-n+1* 


Theorem  2.2:  Let  T  be  an  n-stage  regular  SDTS,  L^ ,  L2  Ln+1  be  its 

input  language,  1st,  nth  output  languages  respectively.  Then 

1 „  is  a  CFPL .  The  proof  is  similar  to  that  of  Theorem  2.1 
12  •••  n+1 

•  *1  i 

except  that  for  the  present  case,  N  =  U  (A  I  A  e  N>  U  <S>,  and  for 

i=1 

each  production  in  T,  there  are  n+1  corresponding  productions  in  G.  CL^ 
is  a  context-free  language.) 
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Theorem  2-3:  Let  T  be  d  simple  SDTS,  ,  L^  be  its  input  dncs  output 
languages  respectively.  Then  L^  is  a  CFPL.  CL^  is  a  context-free 
language . ) 


Proof:  We  shawl  c  onstruct  a  CFPG  G  such  that  Lie  )=L^.  Since  T  is  a 
simple  SOTS,  each  production  in  T  is  of  the  form  A  a,  is  where 
A  t~  N,  tt  c  (NUl1)*,  Sc  (NUA)  *,  a,  0  have  the  same  nonterminals  with  the 
same  permutation.  G  (N  '  ,1*  ,J  ,P, S)  is  constructed  as  follows: 

Step  1.  N '  =  I Af N>  U  ikC | AeN>  U  {S>. 

Step  ?.  1'*  =1'  U  A. 

Step  3.  (1)  Add  S  *  S^S^  <K } K  is  the  label  of  production  beginning 

with  S^>  <p  to  P. 


(2)  For  each  production  A  ^^A^  ...  B^A^  ••• 

*  ★ 

SAB  .  in  R.  where  A.eN,  a.c  I  ,  BeA  ,  AeN,  ada 
n  n  n+1  '  i  r  i  i 

11  1 

A  -►  ...  anAnan+i  lK\K  is  the  label  cf  proauction 

AJ  -  8,A?  ...  A^n.,  >  . 

2  2  2 

and  A  ^  S.A.  ...SAB  „  <K|K  is  the  label  of  productions 
11  n  n  n+1 

beginning  with  the  leftmost  nonterminal  of  the  present 


derivation  string>  4> 


Strictly  speaking,  there  is  little  restriction  on  the  next 
step  of  a  derivation  after  one  application  of  the  production 
beginning  with  a  nonterminal  of  the  form  Au  where  A  is  in  N. 
The  only  restriction  is  that  the  next  applied  production  must 
begin  with  a  nonterminal  of  the  form  A^  where  A  is  in  N. 


The  proof  that  L(G)=L^  is  similar  to  that  of  Theorem  2.1. 
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Corollary  Let  T  be  an  n-stage  simple  SDTS,  be  its  input 

language,  and  L^,  _  Ln+^  be  its  1st,  2nd  —  nth  output  languages 

respectively.  Then  L  is  a  CFPL. 

12.. .n+1 

Rosenkrantz  C 53 J  proposed  the  concept  ot  pr  grammed  grammars  as  a 

new  device  for  generating  languages.  It  is  shown  that  the  class  of  CFPL 

properly  contains  the  class  of  context-free  languages  ana  is  properly 

contained  within  the  class  of  context-sensitive  languages.  Writing  a 

CFP6  is  very  much  like  writing  a  computer  program  an a  is  a  rather 

straightforward  logical  process  compared  with  the  task  of  writing  a 

context-sensi t i ve  grammar.  Swain  and  Fu  L 54 3  further  applied  CFPG  to 

pattern  recognition  and  proposed  a  stochastic  syntax  analyzer  for  CFPL. 

Lu  and  Fu  CUT  then  proposed  an  er ror-cor rec t i ng  parsing  algorithms  tor 

CFPL.  By  the  previous  discussions,  it  is  known  that  a  class  of  TV 

patterns  can  be  described  as  a  CFPL.  Then  the  analysis  of  TV  patterns 

becomes  the  parsing  of  a  CFPL  which  has  been  well  developed  by  Fu  et  at. 

This  fact  is  to  say  that  some  TV  pattern  analysis  problems  can  be 

thought  of  as  parsing  of  a  CFPL.  In  other  words,  we  can  infer  a  CFPG  to 

interpret  or  to  generate  the  time-varying  process.  A  CFPL  parser  can  be 

used  to  classify  an  input  pattern  sequence.  Actually  the  parsing  task 

can  be  performed  by  directly  building  a  parsing  algorithm  from  the 

translation  schema  instead  of  constructing  a  CFPG.  This  parser  consists 

of  n  interrelated  "sub-parser"  each  of  which  deals  with  the 

corresponding  x^  input  string.  The  basic  idea  is  that  these  n  input 

strings  x  ...,x  need  not  be  concatenated.  However  the  relation  among 
1 ,  n 

these  subparsers  still  has  the  spirit  of  a  CFPL  parser. 
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In  applying  syntactic  methods  to  pattern  recognition,  one 
dimensional  (string)  gi ammars  are  sometimes  inefficient  in  describing 
high-dimensional  patterns.  For  the  purpose  of  effectively  Gescribing 
h  i  gh-di  mens  iona  l  patterns,  Fu  and  Bhurgava  C  5  5  li  introduced  the 
application  ot  tree  systems  to  pattern  recognition.  Lu  and  Fu  LltJ 
proposed  er ror-correct ing  tree  automata  for  the  recognition  of  noisy 
patterns.  Five  types  of  error  t ransf ormat ion  are  introduced: 

(1)  the  subst4tution  of  the  label  of  a  node  by  another  terminal  symbol, 

(2)  the  insertion  of  an  extraneous  labeled  node  between  a  node  ana  its 
immediate  predecessor, 

(3)  the  insertion  of  an  extraneous  labeled  node  to  the  left  of  all  the 
immediate  successors  of  a  node, 

(4)  the  insertion  of  an  extraneous  Labeled  .iode  to  the  right  of  a  noce, 

(5)  the  deletion  ot  a  node  of  rank  1  or  0. 

The  three  operations  of  insertions  in  rule  (2),  (3)  and  (4)  are 
named  as  stretch,  branch,  and  split,  respectively,  according  to  the 
relative  position  of  the  inserted  node  to  the  original  tree. 
Apparently,  the  inverse  operation  of  any  type  of  insertion  is  deletion, 
and  the  inverse  of  deletion  operation  is  one  of  the  three  types  of 
i nsert i on  . 


The  deformation  from 

a  pure 

pat 

tern 

to  its  noisy 

pattern  can  be 

oerformed  through  a 

sequence 

of 

app  li cation 

of 

these  error 

transformations.  Through 

the 

use 

of 

the  deformation 

model,  the 

transf ormat ion  concept  can  also  be  applied  to  tree  grammar  to  solve  some 
time-varying  oattern  analysis  problems  which  are  not  easy  to  handle  with 
string  grammar  t rans la t ions .  After  introducing  a  brief  definition,  we 
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propose  a  tree  grammar  translation  schema.  (For  detailed  description 
about  tree  grammar,  refer  to  L123). 

Definition  2.11  :  A  tree  grammar  =  (V,r,P,S)  over  <£  r>  is  expansive 
iff  each  production  in  P  is  of  the  form 

x 

Xn  -  /  \  or  Xn  -  x  where  xe£. 

u  y  y  u 

1  r  Cx ) 

and  Xn,  X„,...,X  ,  .  are  nonterminal  symbois. 

0'  r  '  r(x)  7 

Definition  2.12:  A  tree  SDTS  T  is  a  4-tuple  (V,r,R,S)  where  V,r,$  nave 
the  same  meaning  as  for  a  tree  grammar,  R  is  a  set  of  productions  of  the 
form:  A  ♦  a,  0  such  that  a  and  3  nave  the  same  nonterminals,  B  is  tne 
result  of  application  of  a  sequence  of  error  transformations  to  a  ana 
the  input  grammar  is  in  the  expansive  form. 

An  n-stage  tree  SDTS  can  be  formed  with  each  production  being  of 
the  form  A  -*■  a,  where  each  3-  is  the  result  of  some 

deformation  of  a.  The  concept  of  "programmedness"  can  also  be  applied 
to  tree  grammar.  Consequently,  Lu  and  Fu's  tree  language  analysis 
method  can  be  used  to  analyze  some  more  complex  TV  patterns. 

Z  .4_  Stochast  i  c  Translation 

Definition  2.13:  A  stochastic  syntax-di rected  translation  schema  (SSDTS) 
C83D  is  a  5-tuple  T  =  (N,£,a,R,S)  where 
N  =  a  finite  set  of  nonterminals; 

£=  a  finite  input  alphabet; 

A  =  a  finite  output  alphabet; 
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R  =  a  finite  set  of  rules  of  the  form  p:  A  ->  a, 8  for  A  in  N,  a  in 
(N  U  21  )*,  8  in  (N  U  A)*,  0  <  p  _<  1 ,  with  the  nonterminals  in  ?. 
being  a  permutation  of  those  in  B; 

S  in  N  ~  the  starting  symcol. 

Each  nonterminal  <n  the  a  portion  of  a  rule  has  an  identical,  associated 
one  in  3  (matching  superscripts  are  used  as  needed  for  repeated 
nonterminals) . 

Def i ni t i on  2,14:  A  schema  T  is: 

(a)  unrestricteo  if  for  each  rule  p:  A  +  a,B  the  probability  p  is  not 
conditioned  on  other  rules  or  events. 

(b)  proper  if  for  each  nonterminal  A  the  probaoi l i t i es  of  all  rules  in 
which  A  is  the  leftside  nonterminal  sum  to  1. 

We  consider  only  unrestricted,  proper  schema.  Translations  from  21*  to 

★ 

A  are  produced  as  follows. 

Definition  2.15:  A  translation  form  of  T  is  defined  recursively: 

(a)  1:  ($,S)  is  a  form  with  associated  S*s; 

(b)  if  p:  (wAcr,YA6)  is  a  form  with  the  A’s  associated  and  p:  A  -►  a, 8 
is  a  rule,  then  pp:  (u>aa,y8<$)  is  a  form. 

Definition  2.16:  The  stochastic  translation  from  to  A*  produced  by  T 
with  starting  symbol  S  is  the  set 

t(T,S)  =  {(x,y,p))x  in  /  y  in  A* ,p  =  z  p  for  all  translation  forms  p:  (x,y)> 

T  defines  a  function  p^:  £  xA*  -*•  CO, 13  such  that 
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n  f  (i ) 

p  <x,y)  =  1  n  P-j^^y) 

i=1  j=1  J 

where  there  are  n  distinct  standard  derivations  (e.g.,  leftmost,)  of  the 
translation  form  (x,y)  with  f(i)  steps  in  the  ith  one,  and  p.^(x,y)  is 
the  probability  assigned  to  the  jth  rule  of  the  ith  derivation. 

Definition  2.17:  A  schema  T  is: 

(a)  consistent  if 

l  I  Py(x/y)  =  1. 
x  y 

(b)  simple  if  in  each  rule  p:  A  +  a,B  the  nonterminals  in  a  occur  in 
the  same  order  as  their  associates  in  £ . 

(c)  regular  if  all  rules  have  the  form  p:  A  x8,  yB  or  p:  A  >  x,y 

for  A,  B  in  N,  X  in  £,  y  in  A*. 


Example  2.2:  The  occurrence  of  substitution  error  in  a  4-bit  binary  coae 
transmission  can  be  described  as  follows: 


R  : 


T  =  ({S,A,B,C>,  {0,1>,  {0,1 >,  R,S) 

1  p 

S  ^  0  A,  0  A 

JL  p 

S  0  A,  1  A 

I  Pc 

S  j—t  1  A,  1  A 
I  Ps 

S  1  A,  0  A 

j  P 

A  ^ — -  0  B,  0  B 

—  P 

A  -|= — -  0  B,  1  B 


1  8,  1  B 
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A 

B 

B 

B 

B 

C 

C 

c 

c 


2  Ps 

K 

K 

K 

K 

hi 
i  ps 
jp 

K 


IB,  OB 
0  C,  0  c 
0  C,  1  c 
1  C,  1  c 
i  c,  o  c 
0,  0 
0,  1 
1,  1 
1,  0 


where  is  the  probability  of  correct  transmission  of  each  individual 
bit,  is  the  probability  of  substitution  error  for  each  individual  bit 
and  Pc  +  Pc  =  1,  The  translation  pair  (0000,  0010)  is  derived  as 

follows:^ 

•p  P 

S  4 — -  0  A,  0  A 

i  pc 

4 - -  00B,  00B 

7  PS 

4 — -  oooc,  001C 

7  P 

- — -  0000,  0010 


The 


probabi l i  ty 


of 


this 


deri vat  ion 


1  1  1  1  13 

4  P  x  4  P  x4P  x  4  P  =4rpp- 

2  c  2c  2s  2c  16  c  s 

There  is  another  way  to  describe  the  probabilistic  property 
stochastic  SDTS.  Suppose  that  A  is  a  nonterminal  of  a  stochastic 
and  there  are  n  translation  rules  with  A  +  a  as  leftside  (or  input 
as  defined  before) : 


i  s 


of  a 
SDTS 
ru  le 


A  —  a, 


k 
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37 


A 


2 

2 


C 


2 

2 


C 


1  B  , 

0  C  , 

1  c  , 

0  , 

1  , 


1  B 
1  B 
0  B 
C  C 
1  C 
1  C 
0  C 
0 
1 
1 
0 


The  left  part  above  is  the  set  of  rules  for  input  language  and  each  rule 

is  followed  by  a  set  of  rules  f(j>r  output  Language.  Th^  input  string 

0000  is  derived  as  follows:  S-OA  -  00B  -  000C  -0000.  The 

probability  of  this  derivation  is  — .  Given  this  derivation,  the  output 

P  P  P 

string  0010  is  derived  as  follows  S  —  0  A  —  DOB  —  001C  —  0010.  The 
conditional  probability  of  the  output  string  0010  given  that  0000  is  the 
input  string  is  P^P$.  probability  of  the  translation  pen  r 

<0000,001 0>  is  4t  P3f>  • 

16  c  s 

Def ini t ion  2.18:  A  stochastic  programmed  grammar  L 1 0 J  is  a  5-tuple 
G  =  <WJ'P'S)'  where 

(1)  is  a  finite  set  of  nonterminals; 

(2)  is  a  finite  set  of  terminals; 

(3)  J  is  a  finite  set  of  production  labels; 


(4)  P  is  a  finite  set  of  productions; 
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(5)  S  is  the  start  symbol,  S  e  V... 

N 

Each  production  in  P  is  of  the  form 

(r)  <0  +  Ti  S(U)P(U)  F(W)PCW). 

The  core  of  the  production  is  "w  r\/*  where 

“  c  <vTU  V*Vvt  UvN)* 

and 

n  e  (VTUv/. 

Each  stochastic  production  in  P  has  a  distinct  Label,  r  c  J.  U  e  ana 

W  e  2J  are  called  the  success  and  the  failure  "go-to"  fields, 

respectively.  P(U)  and  P(W)  are  probability  distributions  associated 

with  the  sets  U  and  W,  respectively. 

In  applying  the  stochastic  production  to  an  intermediate  string  £ 

derived  from  S  under  leftmost  interpretation,  if  £  contains  the 

substring  tu,  then  the  leftmost  occurrence  of  <o  is  expanded  into  n  and 

the  next  production  to  be  appLied  is  selected  from  the  success  "go-to" 

field  U  according  to  a  probability  distribution  P(U).  If  £  does  not 

contain  u>,  then  £  is  not  changed  and  the  next  production  is  selected 

from  the  failure  "go-to”  field  with  a  probability  distribution  P(W). 

r1  r2  rn 

Suppose  that  S  =  — >  u  — >  ...  — >  =  x  is  a  derivation  of 

l  c  n+ 1 

x,  where  r.  denotes  the  label  of  the  production  used  to  directly  derive 
from  UK.  The  probability,  p.(x),  associated  with  the  i-th 

derivation  of  x  is  defined  as  the  product  of  the  conditional 

probabilities  p(r1 )p(r2 1 r1 ) . . .p(rn | r ^ ) .  The  interpretat ion  of  the 
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conditional  probability  P^rj  the  probability  of  selecting  r ^  as 

the  next  production  to  be  applied  from  the  success  Hgo-to"  field  if  the 
rj_-|~th  production  is  successfully  applied.  Otherwise,  pCr ^  | r j  — -j > 

denotes  the  probability  of  selecting  as  the  next  production  to  be 
applied  from  the  failure  "go-to"  field  of  the  r^-th  production,  p(r^) 
is  assumed  to  be  1. 

Definition  2.19:  A  stochastic  programmed  grammar  G  is  said  to  oe  a 
stochastic  context-sensitive,  context-free,  or  regular  programmed 

grammar,  if  G  is  a  context-sensitive,  context-free  or  regular  pg. 

The  stochastic  language  generated  by  a  stochastic  pg  G  is 

*  piCx) 

LCG)  =  T(x,p(x))|x  e  \l  s  - >  x,  for  i=  1,...,k 

and 

k 

Z  p . (x)  -  p(x)>, 
i=1 

and  k  is  the  number  of  all  distinctly  different  derivations  of  x  from  S 

defined  in  G  and  p^(x)  is  the  prr  hility  associated  with  the  i-th 

derivation  of  x. 

The  stochastic  language  LCG)  is  called  a  stochastic  programmed 
language.  G  is  said  to  be  consistent  if  and  only  if 

l  P  C  x )  =  1. 

xel(G) 
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Theorem  j2.4_:  Let  and  be  the  input  Language  ana  the  output 

Language,  respectively,  of  a  stochastic  reguLar  transLation  schema 
T(N,]C,A,R,S),  then  is  a  stochastic  context-free  programmed  Language 
(SCFPL).  (L^  is  a  context-free  language.) 

Proof :  We  shall  prove  the  theorem  by  constructing  a  SCFPG  G  such  that 
L(G)  =  L^.  A  SCFPG  G(N *,  2,  ,J,P,S>  is  constructed  as  follows: 

Step  1.  N'  =  ■CA1|AeN>U'CA2|AeN>  UtS>. 

Step  2.  £  =  5Z  (J  A. 

Step  3.  Since  T  is  a  stochastic  regular  transLation  schema,  each 
production  in  R  is  of  the  form 

P:  A  **•  aB,  aB 


P:  A  -*•  a,  a  where  A,BcN,  aeJ^aeA*.  0  _<  P  <_  1 


(1)  Put  all  rules  of  R  which  have  the  same  Leftside  nonterminal  in 
a  group  within  which  all  rules  with  the  same  input  rule  are  further 
grouped  together  as  follows: 


pn:  A  aiBi'  aiiBi 

^12"  ^  a12^1 


P1n(1):  A  **  3 ainMlB 


TV  In  (1 )  1 


P21:  A  a28?'  a21  B2 


P^  ,  v :  A  +  a  8  ,  a  ,  ,B 
mn Cm)  m  m'  ■ 


m  m'  mn(m)  m 
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or 

Pmn(m)*  A  am'  amn(m) 

where  0  <  P .  .  <  1 . 

-  iJ  - 

A,  B.  r  N, 

ai  6  Er 
★ 

a.  .  t  A  , 

1  J 

n(i)  is  the  number  of  rules  having  the 
same  input  rule  A  -►  or 

A  -  a., 

and  m  is  the  number  of  different  input  rules 
for  nonterminal  A, 

1  _<  i  <  m,  1  <  j  <  n(i). 

(2)  For  each  rule  P..:  A  +  a.B.,a..B.  of  the  group  formed  for  A 

i  j  i  i '  i  j  i 

in  (1),  add  the  rule  to  P: 

y  A2  -  a.  .  B2  S(U)  P(U)  4>  1 

l  J  l 

where  y  is  the  assigned  label  for  this  new  rule,  S((J)  and  PCU)  are  the 
success  field  and  the  associated  probability  distributions.  The 
construction  of  S(U)  and  P(U)  is  explained  by  the  following  example: 
Suppose  that  =  A  and  the  set  of  rules  in  R  is  grouped  as  shown  in 
(1).  Then  S(U)  =  Tyd  ),y(2>,. .  .,y(m)>  and  P(U)  =  iPl 'P2' ' ' m 'Pn\}  where 
y(k)  is  the  label  for  the  corresponding  rule  of  the  k-th  input  rule  of 
A,  which  will  be  described  in  (3),  and  P^  =  P^  +  +  ...  +  Pin(-j)  i s 

the  corresponding  probability. 

(3)  For  each  different  input  rule  A  ♦  a.B.  of  the  group  formed  for 
A  in  (1),  add  the  rule  to  P: 


kl 


Y  A1  -  a.B1  S(U>  P(U)  <n  1 

i  i 


where  y  is  the  assigned  Label  for  the  rule,  SCU)  ana  P(U)  are  tne 
success  field  and  the  associated  probability  distribution  respectively. 
SCU)  =  <Y^1^/Yt2)#.../Y(n(i))>,  where  n(i)  is  the  number  of  translation 
rules  with  the  same  input  rule  A  +  a.B^  and  is  the  label  of  the 
rule  in  P  corresponding  to  the  k-th  of  these  translation  rules. 

P(U)  =  ^p1/p2'--'Pn(i)>  wnere 


Pk  =  Pjk/tPi1*Pi2***'+Pin(i)>'  1  -  k  -  n<l)* 

(4)  For  each  rule  ^ ;  A  +  a-,a.  .  of  the  group  formed  for  A  in 

CD,  add  the  rule  to  P: 


Y  A  ^  a*jj  ^  ^  ^  ^ 

where  y  is  the  label  assigned  to  this  new  rule. 

(5)  For  each  different  input  rule  A  +  a.  of  the  group  formed  in 
CD,  add  the  rule  to  P: 


Y  A  ♦  a.  SCU)  PCU)  <t>  1 


where  y  is  the  label  assigned  for  this  new  rule,  SCU)  and  PCU)  are  the 
same  as  defined  in  (3). 

(6)  Add  the  rule  to  P: 


1  S  *  SfS 


1*2 


SCU)  PCU)  4>  1 


where 

1  is  the  label 

assigned  for  this 

new 

ru  le  , 

S(U)  = 

^y(D,y(2),.  ..,Y(m)>  and 

POJ)  =  IP., p,....,P  >, 

1  c'  m 

where  m 

i  s  the 

number 

of  different  input  rules 

with  leftside  nonterminal 

S,  yCD 

i  s  the 

label 

for  the  corresponding 

rule  in  P  of  the  i-th 

input 

rule. 

A3 


P.  =P +P .~+.. .+P ,  n(i)  and  P. .  are  as  defined  (1). 

i  ill?  in(i )'  1 3 

Now  we  proved  that  =  L (G) . 

Let  (xy,P)  be  in  L  .  Then  there  exists  a  set  of  different 
derivation  sequences.  Consider  one  derivation  sequence: 


P1  P?  Pn 

CS/S)  —  (a^  A1,a^A^)  —  (a^  a^A2,a^  cx^A^) . . —  (a^  a-> .  *  an'ai  *  *  an^  = 

From  the  above  construction  procedure  of  the  SCFPG  G,  it  is  known  that 
for  each  derivation  step 


P. 

<aia2**‘aiAi'a1o2*"0tiAi)  (aia2-”ai  +  1Ai+1'a1a2" 
there  exist  the  production  rules  in  P: 

Y  a]  -  ai+1A1i+1  S(U>  PCU)  *  1 

y '  A?  -  “i+1A^+1  S(U')  P(U')  $  1 


%+1Ai+1 


where  S(U)  contains  y'  and  the  corresponding  probability  in  P(U)  is 
P_./q_.  for  some  value  q-.  Besides  there  exists  a  production  rule  in  P; 

y"  kC  .  -  a  A2  S(U")  P(U'  ')  *  1 

1-1  11 

where  SCI)11'  contains  y  and  the  corresoondi ng  probability  in  PCU'*)  is 

V  (pi/qi)  *  qi  =  V 

Therefore  there  exists  a  derivation  in  G^ 


s  1  S  V  fll 


.1S2  P1/Q1  1  -  - 
a1A1S  1A1a1A1 


.2  q2 
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V2“-ai  +  1Ai+1alV"ai+1Ai+1 


qi+2 


P  /c i 
n  n 


a1a2’ * 'ana1a2' ' *an  xy  ' 


q1#(P1/q1)'q2*(P2/q2), 


.qi-cpi/qi>...qn-(pn/qn> 


S  P  *p  P  p 

1  K2*  *  * "  *  n 


Therefore,  for  each  derivation  of  xy  in  T,  there  exists  a  unique 
corresponding  derivation  with  the  same  probability  in  G.  Besides,  two 
different  derivations  of  xy  in  T  will  not  map  to  the  same  derivation  in 
G.  (In  other  words,  it  is  a  one-to-one  correspondence).  Similarly,  for 
each  derivation  of  xy  in  G,  there  exists  a  unique  corresponding 
derivation  with  same  probability  in  T  (also  a  one-to-one 
correspondence).  Therefore,  there  exists  a  one-to-one  and  onto 
relationship  between  the  set  of  derivations  of  xy  in  T  and  the  set  of 
derivations  of  xy  in  G.  Therefore, 

(xy,P)  e  L12  <==>  (xy,P)  e  L(G)  . 

L12  =  L(G)  . 


Theorem  2,5:  Let  and  L2  be  the  input  language  ana  the  output 
language,  respectively,  of  a  stochastic  simple  syntax-di rec ted 
translation  schema  SSSOTS  T(N, S,A,R,S),  then  is  a  SCFPL. 


i 


1 
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Proof:  We  shall  construct  a  SCFPG  G  such  that  L(G)  =  L„  .  A  SCFPG 
-  ,  ^ 

t 

G (N  * ,  Z  ,P,S)  is  constructed  as  follows: 

Step  1.  N*  =  -CA1  [ A  e  N>  U  CA2|A  e  N>  U  CS> 

Step  2.  Z  =  Z  IJ  A. 

Step  3-  Since  T  is  a  SSSDTS,  each  rule  in  T  is  of  the  form 
P:  A  a, 3 

where  0  _<  P  _<  1,  A  e  N,  ae  (N  U  Z  ^  :  B  e  (NUA)  ,  and  a,B  have 
the  same  nonterminals  with  the  same  permutation. 

(1)  Put  all  rules  of  R  which  have  the  same  Leftside  nonterminal  in 
a  group  within  which  all  rules  with  the  same  i nput  rule  are  further 
grouped  together.  The  process  is  demonstratea  by  an  example  for 
nonterminal  A  as  follows: 


P11 :  A  -  a1'  S11 

P12:  A  *  V  812 


P1n (1 ) "  A  -  V  61n(1) 


P  •  A 
21-  . 


a2'  B21 


Pmn(m)'  A  8mn(m) 

where  0  <  P .  .  <1,  a.  e  (N  U  ZK  B .  .  c  (NUA)  ,  a .  ,  6  •  -  have  the 

—  1 j  —  1  TJ  1  1  J 

same  nonterminals  with  the  same  permutation,  1  <  j  n(i),  1  <  i  <_  m, 
n(i)  is  the  number  of  rules  having  the  same  input  rule  A  ♦  a  m  is  the 
number  of  different  input  rules  for  nonterminal  A. 

(2)  For  each  rule  P..:  A  a of  the  group  formed  for 

i  j  i '  i  j  3  ^ 

nonterminal  A  in  (1),  Where  a.  =  a. .A. ...a.  A  a.,  , .  w 

'  i  ill  in  n  i  (n  +  1 ) ' 


8ir6n1A1- 


f ® i j  (n  +  1  )  ' 


J 

.  3 


i 

t 

! 


4 
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a .  e 

i  k 


£  '  6ijk  e  A  /  Ak  e  M,  1  f  k  <  n+1,  n  >  1, 


add  the  rule  to  P: 

>  a2  *  s<u>  P(u>  *  1 

where  y  is  the  Label  assigned  for  this  new  role, 
SCU)  =  {y(1),y(2),...,y(m)>,  PCU)  =  iP1,P2,.../P  },  y(k)  is  tne  label 
for  the  corresponding  rule  of  the  k-th  input  rule  of  nonterminal  , 

P^  =  P|C-|+P|<2*  •  •+Pkn(k)'  ^  k  the  number  different  input 

rules  of  p^£  is  the  probability  of  the  £~th  translation  rule  of  the 

k-th  input  rule  for  A^,  n(k)  is  the  number  of  translation  rules  of  the 
k-th  input  rule  for  A^. 

(3)  For  each  different  input  rule  A  -►  a_.  of  the  group  formed  for  A 
in  If)/  where  -  Oli0,^1"""ai  (n-1)^n0!in/  ai  k  £  wC  '  ^k  £  ^  ^  ^ 


n  >  0, 


add  the  rule  to  P: 


y  A1  -  a.nA]...a,  ,  ,Va.  SCU)  PCU)  <l>  1 

1 0  1  i  (n-1 )  n  in 


where  y  is  the  label  assigned  for  this  new  rule, 
S(U)  =  {y  (1 ) ,y (2) , . . • y (n (i ) ) >,  PCU)  =  iP^ ,P-,, . . „ ,P^ >>,  n(i)  is  the 
number  of  translation  rules  with  the  same  input  rule  A  +  c^,  ylk)  is  the 
label  of  the  corresponding  rule  in  P  of  the  k-th  translation  rule  with 

input  rule  A  ♦  cu,  P|c  =  pik  /  <Pi1+P12+-*-+PinCi))'  Pik  15  the 
probability  of  the  k-th  translation  rule,  1  _<  k  <_  nii). 

(4)  For  each  rule  P..:  A  -+  a._£..  of  the  group  formed  for  A  in 

i  J  l  '  i  j  3 

(1),  where  a.  e  V  ,6. .  e  A  , 

'  i  ^  '  i  l  ' 
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add  the  rule  to  P: 

Y  A2  -  S.j  S(U)  P(U)  $  1 

where  v  is  the  ^afcel  assigned  for  this  new  rv !  , 

S(U)  =  <Y(1/1),Y(1,2)/,...,Y(1,m(1)),Y(2r1)...Y(t,m(t))> 


P(U)  =  *P1,1'P1,2',**'P1,m(1)'P2,1'***'Pt,m(t)>'  where 

t  is  the  number  of  nonterminals  in  N,  m(k)  is  the  number  of  different 
input  rules  for  the  k-th  nonterminal,  y(k,£.)  is  the  label  of  tne  rule 
added  in  (3)  corresponding  to  the  £-th  input  rule  of  the  K-th 

nonterminal  of  H,  pk/t  =  Pk,t/1+Pk/i/2+-+pk/l,n(k/U'  n(k'U  15 
number  of  translation  rules  associated  with  the  fc-th  input  rule  of  the 

k-th  nonterminal,  Pk^2 .  Pk/i,n(k,*)  are  :neir 

cor  respond! ng  probabilities. 

(5)  Add  the  rule  to  P; 


1  o2 


5  S 


S(U)  P(U) 


where  y  is  the  label  assigned  for  this  new  rule. 


S  (U )  =  *Cy(1),y(2),  —  ,y(m)>,  P  (U )  -  _ 'Pm^  '  where 

m  is  the  number  of  different  input  rules  for  nonterminal  S,  y(i)  is  the 

label  for  the  rule  added  in  (3)  corresponding  to  the  i-th  input  rule  of 

S,  P.  =  P .. +P .«+•• *+P .  n(i)  is  the  number  of  translation  rules 

'  i  i 1  i2  in  (i ) ' 

associated  with  the  i-th  input  rule  of  S,  P ..  ,P ,P .  ...  are  the 

1 1;  i2;  '  m(i ) 

correspond! ng  probabilities  of  these  translation  rules. 
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The  proof  of  l(G)  =  is  similar  to  that  of  theorem  2. 4. 


2.5  Illustrative  Examples 


Example  2.3:  The  following  CFPG  G  generates  the  Language 


LSQ={anbncndn|n>1> 


which  could  be  interpreted  as  the  language  of  squares  of  side  length  n  = 


1,2,. . L10J 


6SQ  =  ^N'VT/J/P,S^ 


where  the  vocabulary  consists  of 


VN  =  tS,A,B,C,D> 
a  c 

VT  =  {-,  b  +  ,  «■,  dO 


the  label  set  is 


J  =  {1,2, 3, 4, 5, 6, 7>, 


and  the  production  set  P  consists  of  7  rules: 
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Success 

Fai lure 

Label 

Core 

branch 

branch 

1 

S 

-*■ 

aAB 

T2,3> 

<*> 

2 

A 

- 

aAC 

L2,3> 

❖ 

3 

A 

D 

{4} 

4> 

4 

C 

- 

o 

C5> 

5 

D 

ODc 

L6> 

V 

6 

B 

-► 

d 

C  7} 

$ 

7 

D 

be 

$ 

4> 

However,  from  a  different  point  of  view,  this  grammar  can  also 
describe  the  following  TV  pattern:  i (x^ ,x^,x^,x^ ) | x.  is  a  line  segment, 
x.  leads  x .  „  by  90°,  and  each  x.  has  the  same  length.} 

Example  2.4:  Zucker  L56]  used  the  concept  of  t ransf ormat iona l  grammar  to 
analyze  texture.  He  first  built  an  ideal  texture.  Then  a  set  ot 
t ransf ormat ion  rules  were  used  to  transform  the  ideal  texture  into  a 
real  texture.  Using  translation  schema,  we  can  nandle  these  problems 
too.  For  example,  consider  the  pattern  shown  in  Fig.  2.1.  A  2-stage 
tree  SDTS  with  substitution  error  only  can  be  used  to  characterize  such 
a  time-varying  image  sequence.  For  an  iteratively  varying  pattern  (like 
TV  texture),  an  n-stage  tree  SDTS  can  be  constructed  to  model  its 
evolving  process. 

Examp le  2.5:  With  the  development  of  CT  (Computerized  Tomography) 
technique,  a  3-D  object  is  usually  represented  as  a  sequence  of  slices 
(e.g.  X-Y  plane  cross-sections  along  Z-axis).  The  translation  schema 
can  be  applied  to  describe  these  3-D  objects.  Let  ,x^, - . . x^>  be  a 
sequence  of  X-Y  plane  cross-sec t ions  with  Z-coordinate  being  1,2,. ..n, 
an  n-stage  SDTS  can  be  constructed  and  then  transformed  to  a  CFPG.  Here 
a  simple  example  is  given  for  an  object  described  by  4  slices  (Fig. 
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This  object  is  described  by  the  following  "programmed"  SDTS 
T({S,A,B,C,D,E,F>,£,£,  J,  R,  S) 


where  l  a  /  ,  b  |  ,  c  \  ,  d  -  }.  R  contains 

Labei.  Core  Success 

Field 


1 

S-A3CDEF  ,ABCDEF  ,ABCDEF 

,ABCDEF 

C2,3> 

2 

A-*d 

,d 

,d 

,d 

3 

A^dA 

,dA 

,dA 

,dA 

C5> 

4 

B+a 

,ab 

,ab 

C6> 

5 

&>aB 

,aBb 

,aBb 

,aB 

C7> 

6 

C-c 

/C 

,c 

C8> 

7 

C>cC 

,cC 

,cC 

,cC 

C9> 

8 

D-d 

,d 

,d 

,d 

C10> 

9 

D-dD 

,dD 

,dD 

,dD 

cm 

10 

E-a 

,ab 

,ab 

C12> 

11 

F-aE 

,aEb 

,aEb 

,aE 

C13> 

12 

F+c 

,c 

/  c 

/C 

♦ 

13 

F-cF 

,cF 

,cF 

,cF 

C2,3> 

Examp 

Le  2*6: 

Here  an 

example  is  given 

to  show  the 

transformation  of  an 

SDTS 

into  a 

CFPG.  The  transf ormat i on 

procedure  was  described  in  theorem 

2.3. 

Suppose 

that  an 

SDTS  is  given 

as 

fol lows : 

z 

ii 

h- 

E, 

£,r 

,S) 

where 

N  =  iS 

,A,B,C,D> 

,  £  =  <  a  , 

b 

✓  C 

,  d 

,  e  >.  R  contains 

S  -  A BCD,  A BCD,  A BCD 
A  +  aA  ,  bA  ,  bA 

A  ♦  a  ,  b  ,  b 

B  -  dB  ,  eB  ,  eB 

B  ♦  cB  ,  eB  ,  eB 

B  >  d  ,  d  ,  d 

C  cC  ,  aC  ,  bC 
C  ^  aC  ,  aC  ,  bC 

C  *  d  ,  a  ,  b 

C  >  dC  ,  aC  ,  bC 

D  ♦  aD  ,  bD  ,  bD 

D  ♦  a  ,  b  ,  b 


e  >.  R  contains 
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A  CFP6  G(N'^,J/P,S)  is  formed  as  follows: 

The  following  steps  follows  the  procedure  of  theorem  2.3. 

Step  1:  N'  =  tA1 |A  e  N>  U  <A2 | A  e  N>  U  <A3 | A  e  NJ-  UiS> 

=  isy  U  <a\b1,C1,d\s1  |i=1,2, 33- 
step  2:  £  = 

12  3 

Step  3;  (1)  Add  S+SSS  (  >  to  P.  The  content  of  the  success  fiela 

will  be  filled  when  all  the  rules  are  constructed. 

(2)  For  the  rule  in  R:  S  +  A  BCD,  A BCD,  ABCD 


add 

,1 


1111 
a'bVd1  {  > 


s2  -  a2b2c2d2  -c  > 
s3  -  a3b3c3d3  i  y 


to  P 


For  the  rule  in  R:  A  -►  aA;  bA,  bA 


add 

A1  -  aA1  l  y 
A2  -  bA2  {  > 
A3  -  bA3  t  > 


to  P 

For  the  rule  in  R:  A  a,b,b 


add 

.1 


a  t  y 


A  ♦  b  {  } 
A3  -  b  i  y 


to  p 


and  similarly  for  the  rest  of  the  rules  of  R.  Then  fill  in  those 
success  fields  as  described  in  theorem  2.3.  Then  for  convenience. 


replace  A^B^C^D1 ,  A2,B2,C2,D2,  A3,B3,C3,D3  by  A,B,C,D,E,F ,G,H,W,X,Y,Z 
respectively.  The  following  is  the  resulting  CFpG. 


N'  =  <s,s1,s2,s\ a,b,c,d,e,f,g,h,w,x,y,z} 

p  = 


Success 

Success 

Label 

Core 

Field 

Label  Core 

Field 

1 

12  3 

S*S 

<2} 

17 

X+d 

{18,19} 

2 

sl+ABCD 

<3> 

18 

C-cC 

{22> 

3 

S^-EFGH 

<4> 

19 

C+aC 

{22} 

4 

S  +WXYZ 

<5} 

20 

C-dC 

{22} 

5 

A+aA 

<7> 

21 

C+d 

{23} 

6 

A+a 

<8> 

22 

G**aG 

{24} 

7 

E+bE 

<9> 

23 

G+a 

{25} 

8 

E-b 

C10> 

24 

Y+bY 

{18,19,20,21} 

9 

W+bW 

<5,6> 

25 

Y+b 

{26} 

10 

W+b 

<11> 

26 

D+aD 

{28> 

11 

B+dB 

<14  > 

27 

D+a 

{29} 

12 

8-cB 

<14> 

28 

H+bH 

{30} 

13 

B-*d 

<15> 

29 

H+b 

{31} 

14 

F+eF 

<16> 

30 

Z-bZ 

{26} 

15 

F^d 

<1  7> 

31 

Z+b 

$ 

16 

X+eX 

<11,12,13} 

Example  2.7:  The  stochastic  translation 

schema  T(N 

,E/A,R,S)  described 

in  Example  2.2 

is  transformed  to 

a  SCFPG 

G(N',  £*, J 

,P,S)  as  follows: 

Step 

1.  N'  = 

-CA1  1  A  e  N>  U  “CA2  |  A 

e  NJ  U 

CS> 

= 

11112  2 
iS  ,A  ,B,C  ,S%A% 

62,C2,S} 

Step 

2.  E’  * 

E  U  A  =  C0,1> 

Step 

3.1.  Put 

the  translation  rules  of 

T  in  the 

order  described  in 

theorem  2.4*  (They  are  already  in  order  in  Example  2.2). 
jP 

Step  3.2.  For  rule  S  — -  OA,  OA 
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add  to  P:  y  S2  ♦  OA2  SCU)  P(U)  4  1 

2? 

For  rule  S  —  OA,  1A 

add  to  P:  T  S2  >  1A2  SCU)  P(U)  4  1 

(y  ,  S(U)  and  P(U)  are  given  Later) 

Other  rules  are  ^dded  similarly.  ^ 

■^■P  -^P 

Step  3.3.  For  the  rules  S  OA,  OA  and  S  — -  OA, 
add  to  P:  y  S1  -  OA1  SCU)  P(U)  4  1 

Other  rules  are.added  similarly. 

4#% 

Step  3.4.  For  the  rule  C  -  0,  1 

add  to  P:  y  1  4>  1  <j>  1 

Other  rules  are  added  similarly. 

jP  yP 

Step  3.5.  For  the  rules  C  -=-£  0,  0  and  C  -=—^  0,  1 
add  to  P:  y  C1  -  0  SCU)  PCU)  4  1 
Other  rules  are  added  similarly. 

Step  3.6.  Add  the  following  rule  to  P: 

1  S  -  S^2  SCU)  PCU)  4  1 

The  final  form  of  the  SCFPG  is 


GC{S1,A1,B1,C1,S2,A2,B2,C2,S>,-C0,1>,J,P,S) 


P: 

Label 

1 

2 

3 

4 


Core 


PCU) 


s  *  s^s2 

{2,3> 

4  ' 

i> 

S1  OA1 

-cio,m> 

<Pc' 

Ps> 

S1  -  1A1 

f10,11> 

<Ps' 

Pc> 

> 
_ 1 

+ 

O 

CD 

a2,i3> 

tP  , 

p  > 

c 

s 

A1  -  IB1 

C12,13> 

<Ps' 

V 

PCW) 

1 

1 

1 

1 


1 A 


5 


1 


**”*^mmm mmmm 
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6 

B1 

oc1 

014,15} 

<P  / 

c' 

V 

4> 

1 

7 

B1 

-► 

•1 

1C 

014,15} 

<P  , 
s' 

V 

4> 

1 

8 

c1 

-+ 

0 

<16,17} 

<p  , 
c' 

p  > 
s 

4> 

1 

9 

-*■ 

1 

<16,17} 

V 

4> 

1 

10 

s2 

-+■ 

OA2 

<4,5} 

4 ' 

¥ 

4> 

1 

11 

s2 

- 

1  A2 

<4,5} 

4 ' 

¥ 

4> 

1 

12 

A2 

-¥ 

OB2 

<6,7} 

4' 

¥ 

4> 

1 

13 

A2 

IB2 

<6,7} 

4 ' 

¥ 

4) 

1 

14 

B2 

-+• 

2 

OC 

<8,9} 

4 ' 

¥ 

4> 

1 

15 

B2 

-* 

1C2 

<8,9} 

4 ' 

¥ 

4> 

1 

16 

c2 

-► 

0 

i 

4> 

1 

17 

c2 

1 

i 

4> 

1 

2.6  Conclusion 


The  time-varying  pattern  analysis  problem  is  investigated  in  this 
chapter.  The  problem  is  analyzed  through  the  use  of  deformation  models 
and  translation  schemas.  It  is  shown  that  some  time-varying  patterns 
can  be  characterized  by  a  context-free  programmed  language.  Therefore, 
the  well  developed  error-correct ing  string  parser  can  be  applied.  The 
proposed  method  can  be  extended  to  problems  involving  three-dimensional 


patterns. 

Tree  translation  is  proposed 

to 

analyze  more 

complex 

TV 

patterns. 

Stochastic  translation  is 

also 

presented  to 

mode  l 

the 

stochastic  properties  of  TV  patterns. 


CHAPTER  3 


A  GENERALIZED  SYNTAX-DIRECTED  TREE  TRANSLATION  MODEL 

Introduction 

In  chapter  2,  we  have  proposed  a  syntactic  method  for  time-varying 
pattern  analysis-  An  input  sequence  x^,  where  is  the 

pattern  representat i on  at  time  t_.,  is  analyzed  through  a  translation 
model  which  is  defined  as  a  mapping  from  a  language  L^  to  another 
language  L^  in  formal  language  theory.  Consider  L^  as  the  set  of 

possible  patterns  occurring  at  time  t^  and  L^  as  the  set  of  patterns  at 
time  t^.  The  relation  governing  the  time-varying  process  of  the 
sequence  is  then  formulated  as  a  translation  problem.  In  this  chapter, 
the  formulation  of  string  translation  is  extended  to  trees  using  a 
generalized  syntax-di rected  model-  The  generalized  model  is  compared 
with  the  conventional  top-down  and  bottom-up  tree  translation  model.  It 
is  shown  that  both  the  top-down  and  the  bottom~up  models  are  special 
cases  of  the  generalized  model.  A  parsing  algorithm  for  this 
generalized  tree  translation  model  is  presented. 

Definitions  and  notations  that  will  be  referred  to  are  briefly 

reviewed  £57,58,59]. 

An  alphabet  £  is  ranked  by  a  function  r:£  ♦  N  which  assigns  a  rank 

-1 

to  each  member  of  £.  For  each  n,  l  =  r  (n)  denotes  the  set  of  symbols 

in  £  which  have  rank  n.  Intuitively,  the  rank  of  a  symbol  is  the  number 
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of  sons  it  has  when  it  Labels  a  node  in  a  tree. 

Let  H  denote  the  set  containing  Left  and  right  brackets  and  comma. 
To  avoid  possible  confusion,  ranked  alphabets  are  not  allowea  to  include 
elements  of  ru  For  a  ranked  alphabet  l,  the  set  l *  of  (finite  labeled) 
trees  over  the  alphabet  Z  is  the  least  set  of  strings  in  (Z  such 

that 

(1)  Z  d  Z  and 

o  -  *' 

(2)  for  n  >  0,  b  e  Z  and  t^,  t^,  ...,  tn  e  l*,  bCt^,  t^,  ..., 

t]  €  Z 
n  * 

Definition  3.1 :  If  I  is  a  ranked  alphabet  and  A  is  an  alphabet,  then 
£  +  (A),  the  set  of  trees  in  Z*  indexed  by  A,  is  defined  recursively  as 
follows . 

1)  I  U  A  C  £  (A) . 

o  -  * 

2)  If  b  e  £  ,  n  >  0,  and  e  £  (A),  then  bLt., _ ,t  j 

n  i  n  *  in 

e  £^(A). 

Definition  3.2:  A  (nondeterminist ic)  top-down  tree  transducer  is  a 
5-tuple  M  =  (Q,  Z,  A,  R,  Q^)  where 

(1)  Q  is  a  finite  set  of  states, 

(2)  I  is  a  finite  ranked  alphabet  called  the  input  alphabet, 

(3)  A  is  a  finite  ranked  alphabet  called  tne  output  alphabet, 

(4)  C  Q  is  a  set  of  starting  states,  ana 

(5)  R  is  a  finite  set  of  rules, 

R  C  U  (Q  x  L  )  X  (A  u  (Q  X  X  ))  . 

-  n>0  n  n  * 
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A  rule  is  written  in  the  form  (q,b)  +  w,  where  a  e  Q,  b  e  Z  ,  ana  w 

'  '  n' 

e(A  U  (Q  x  x  ))  for  some  n,  where  X  denotes  the  set  x  } 

n  *  n  i  c w  n 

for  n  >  0  and  denotes  the  empty  set.  refers  to  the  ith  son  of  the 
current  input  node. 

The  behavior  of  a  top-down  transducer  is  defined  inductively  in 
terms  of  the  output  produced  from  a  tree  starting  in  state  q. 

Definition  3.3:  Let  M  =  (Q,  I,  A,  R,  Q^)  be  a  top-down  tree  transducer, 
and  let  q  e  Q.  For  a  tree  t  e  l*,  the  set  of  trees  output  from  t  by  M 
starting  in  state  q  is  denoted  by  M(q,t)  and  is  defined  inductively  as 
f ol Lows  „ 

(1)  If  t  =  b  e  l  then  M(q,t)  =  *Cw|(q,b)  ■*  w  c  R>; 

(2)  If  t  =  bit.,  ...,  t  j  e  l  then 

M(q,t)  =  U  w(<p,x  •>:M(p/t  . )  |peQ,  1  _<  j  n). 

(q,b)+W£R  **  J 

Definition  3.4:  A  nondetermini st i c  bottom-up  tree  transducer  L583  is  a 
five-tuple  M  =  (Q,  £,  A,  R,  c)  where 

1)  Q  is  a  finite  set  of  states, 

2)  Z  and  A  are  finite  input  and  output  ranked  alphabets, 
respect i ve ly . 

3)  F  Q  is  a  set  of  final  or  accepting  states, 

4)  [flu  :  ifll  =  iflx  =  |>,  and 

5)  R  is  a  finite  set  of  transition  rules  such  that  every  rule  in  R 


1 


i 

/ 


: 


is  either  of  the  form 
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i 


(b)  +  (q,t),  where  b  e  q  e  Q,  ana  t  e  A^, 
or  of  the  form 

(b,qv...,q  )  ♦  (q,t),  where  n  >  0,  b  c  Z  q,  q  ,...,q  £  Q,  and  t  c  a  U  ). 

The  behavior  of  a  bottom-up  transducer  on  an  "input  tree  is  aefined 
inductively  as  follows. 

Definition  3,5:  Let  M  =  (Q,  Z,  A,  R,  F)  be  a  bottom-up  tree  transducer, 

and  let  q  e  Q.  For  a  tree  t  e  Z*,  the  set  of  trees  which  M  can  output 

from  t  ending  in  state  q  is  denotea  by  M(q,t)  ana  is  aefinea  mauctively 
as  follows; 

(1)  For  b  e  M(q,b)  =  Lw|b  -►  (q,w)  e  R>, 

(2)  For  t  =  bCtv...,tn3  e  Z*, 

MCq,t)  =  "Cv  | f or  some  rule  <b/q^.../q  )  -►  (q,w)  e  R, 

some  i,  1  <  i  <  n,  and  same  u .  e  M(q.,t.),  v  e  W(x.;u-)> 

Definition  3.6:  A  generalized  syntax  directed  translation  (GSDT )  L 5/j  is 
a  four-tuple  F  =  (G,  A,  r,  R) ,  where; 

Cl)  G  =  (V,  Z,  P,  S)  is  a  proper  context  free  grammar; 

(2)  A  is  a  finite  set  of  output  symbols; 

(3)  r  is  a  finite  set  of  distinct  translation  symbols  of  the  form 

t.(A),  where  i  is  an  integer  and  A  is  in  V  -  Ls>,  plus  the  symbol  . 
Whenever  it  is  possible  to  do  so  without  confusion,  we  will  denote  t^CA) 
by  A^.  We  call  A^  the  i-th  translation  symbol  associated  with  A* 

(4)  R  is  a  function  which  associates  with  each  production  A  a  in 

P,  a  set  of  semantic  rules  LAJ]  =  3^  ^  =  82,...,Am  =  in  which  each 
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* .  is  a  string  in  (r  U  A)  ,  such  that  all  translation  symbols  appearing 
in  are  translation  symbols  associated  with  nonterminals  appearing  in 

a . 

For  each  x  ir  l  we  detine  fix),  the  ^  of  outputs  of  x  as 
follows : 

(1)  If  x  is  net  in  L(G),  then  f(x)  =  0. 

(2)  If  x  is  in  L(G),  then  each  parse  tree  with  yield  x  defines  an 

element  y  in  F(x),  which  is  the  value  of  the  translation  symbol 
associated  w"th  the  root.  The  value  of  is  computed  bottom-up  as 

follows : 

(i)  With  each  interior  node  N  of  the  parse  tree  labeled  A  a  are 

associated  the  translation  symbols  A^,  A^,  A  ,  which  are  ail  the 

translation  symbols  associated  with  A.  The  values  of  these  translation 
symbols  at  N  are  computed  using  the  semantic  rules  and  the  values  o*  the 
translation  symbols  at  the  descendants  of  N  as  follows. 

★ 

(ii)  Suppose  a  is  x  _  5,  x,  ,  where  x.  is  in  I  ana  b  is 

0^122  k  k'  j  j 

in  Suppose  a;  ~  y^^c^y^  ***  C,yl  the  semant'ic  ruLe 

for  A..,  where  y  is  in  A  and  is  a  translation  symbol  in  r  associated 
with  for  some  1  _<  h  .  <  k .  Then  v(A.),  the  value  of  at  noce  N,  is 

j  ~  :  1  * 

the  string  y^v  (C^)  y^  v  (C^)  y£  •••  v(C^)y^  in  A  ,  where  v(C^)  is  the  value 
of  C.  at  the  descendant  of  N  which  is  labeled  by  a  production. 

3  ■  j 

T(F),  the  translation  defined  by  F,  is  the  set  *C(x,y)iy  e  FCx)>. 

Example  3.1  :  Let  F  =  (G,  tS^,  A^,  ,  85>,  R),  where  the 

productions  of  the  grammar  and  the  associated  semantic  rules  are: 


Product  ions 


Semantic  rules 
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(1)  S  -  A 

(2)  A  -  aAbB 


(3)  A  +  oAa  B 


(4)  B  -  A 


(5)  A  +  E 


F  defines  the  translation  -C(w,a1b1)|i  _>  0  and  w  e  Ta,b}*,  such  that  w 
has  i  a's  and  i  b's>. 

3.2  Generalized  Syntax-Pi  rected  Tree  Trans  (.at  ion 
A  top-down  tree  transducer  CTDTT)  does  not  specify  an  input 
language,  it  assumes  that  all  trees  in  Z*  are  input  sentences.  A 
bottom-up  tree  transducer  (BUTT)  does  specify  an  input  language,  which 
is  a  subset  of  £*,  but  given  an  input  tree  sentence,  even  though  a  BUTT 
can  copy  a  subtree  in  an  output  sentence  many  times,  all  the  copies  of  a 
subtree  are  the  same,  while  a  TDTT  can  generate  different  copies  of  a 
subtree.  There  exist  translations  which  can  be  implemented  by  a  TDTT, 
but  not  by  a  BUTT.  There  also  exist  translations  which  can  be 
implemented  by  a  BUTT,  but  not  by  a  TDTT. 

Example  3^:  An  arithmetic  expression  involving  addition, 

multiplication,  a  constant  c,  and  a  variable  y  may  be  represented  by  a 
tree  over  the  alphabet  z  =  <+,  ★,  y,  c>,  where  +  and  *  have  rank  2  ana  y 
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and  c  have  rank  0.  We  construct  a  deterministic  top-down  transducer  M 
which  takes  the  formal  derivative  with  respect  to  y  of  the  expression 
represented  by  an  input  tree  in  Z*.  Let  A  =  Z  |J  *C1,Q>  where  1  and  D 
have  rank  0.  Let  M  =  (TD,I>,  l,  A,  R,  <0>)  be  a  top-down  transducer, 
where  R  contains  the  following  rules: 

<D,  +  )  -►  +C<D,x1>,<D,x2>J, 

(D,*)  -►  +L*C<D,x^>y<I,x2>],  *C<I,x^  >,<D/x2>]], 

(D,y)  *  1,  (D,c)  ■*  0,  (I, a)  -►  o  for  a  e{y,c>,  and 

(I, a)  *  cC<I,x^>,<I,x2>D  for  o 

The  tree  translation  defined  by  M  cannot  be  defined  by  a  BUTT.  The 
main  reason  is  due  to  the  second  rule  above.  With  this  rule,  the 
subtree  of  an  input  tree  *Cx^,x2D  is  translated  two  times  as  <D,x^> 
and  <I,x/j>  respect i vely.  For  instance. 


+ 


1  c  y  o 


In  the  above  translation,  y  is  translated  two  times  differently  (the 
first  time  as  1  and  the  second  time  as  y).  But  from  the  definition  of 
BUTT,  we  know  that  BUTT  cannot  generate  this  tree  translation  pair.  On 

the  other  hand,  a  translation  which  contains  only  a  single  tree  pair 

★  + 

t(  /\  ,  /\  )>  cannot  be  implemented  by  a  TDTT  because  a  TDTT  cannot 

y  c  r  o 

specify  finite  input  tree  set.  While  this  translation  can  be 
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implemented  by  the  following  BUTT: 

M=(Cq0/rq1/q2>/  {y,c,*>,  C1,0,+>,  R,  {qQ>> 

R=Cy(q1,1)/  c-(q2,0),  (*,<}•,  ,q2)-(q0,  +  LX.J  ,x2j)>. 

Based  on  the  definition  of  GSDT  given  by  Aho  and  Ullman,  we  propose 
a  simitar  definition  for  trees ,  called  the  generalized  syntax-di rec tea 
tree  translation  (GSDTT).  It  can  be  shown  that  both  TDTT  and  BUTT  are 
special  cases  of  GSDTT. 

A  GSDTT  is  defined  as  follows: 

Definition  3.7:  A  generalized  syntax-directed  tree  translation  (GSDTT) 
is  a  four-tuple  F  =  (G,  A,  r,  R) ,  where: 

(1)  G  =  (V,  Z,  P,  S)  is  a  regular  tree  grammar; 

(2)  A  is  a  finite  set  of  output  symbols; 

(3)  r  is  a  finite  set  of  distinct  translation  symbols  of  the  form 

t^(A),  where  i  is  an  integer  and  A  is  in  V  -  -CS>,  plus  the  symbol  . 

whenever  it  is  possible  to  do  so  without  confusion,  we  will  denote  t^CA) 
by  A*.  We  call  A^  the  i-th  translation  symbol  associated  with  A. 

(4)  R  is  a  function  which  associates  with  each  production  A  ■+  a  in 

P,  a  set  of  semantic  rules  =  $  A->  =  A  =  0  >,  in  which 

1  c  c'  r  m  m  ' 

each  0.  is  a  tree  in  (r  U^).,  such  that  all  translation  symbols 

i  * 

appearing  in  are  translation  symbols  associated  with  nonterminals 

appearing  in  a. 

For  each  x  in  2*  we  define  F(x),  the  set  of  outputs  of  x  as 
follows : 

(1)  If  x  is  not  in  L(G) ,  then  F(x)  =  4>* 
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(2)  If  x  is  in  L(G),  then  each  parse  tree  with  yield  x  defines  an 
element  y  in  F(x),  which  is  the  value  of  the  translation  symbol 
associated  with  the  root.  The  value  of  is  computed  bottom-up  as 

f ol  lows : 

(i)  With  each  interior  node  N  of  the  parse  tree  labeled  A  -  a  are 

associated  the  translation  symbols  which  are  all  tne 

translation  symbols  associated  with  A.  The  values  of  these  translation 
symbols  at  N  are  computed  using  the  semantic  rules  and  the  values  of  the 
translation  symbols  at  the  descendants  of  N  as  follows. 

(ii)  Suppose  TB..|0  <  j  <  k  for  some  k>  is  the  set  of  nonterminals 

appearing  in  a,  is  the  semantic  rule  for  A^  and  TCj)1  _<  j  <  k 

for  some  l>  is  the  set  of  translation  symbols  appearing  in  8^  with 
being  the  translation  symboL  associated  with  .  Then  v(A^),  the  value 

j  1 

of  A.,  at  node  N,  is  the  tree  8^  with  each  being  replaced  by  v(C  J, 

where  v(C.)  is  the  value  of  C.  at  the  descendant  of  N  which  is  labeled 
3  J 

by  a  production.  T(F),  the  translation  defined  by  F,  is  the  set 

j 

{ (x,y) | y  z  F (x)>. 

Theorem  3.1:  For  any  BUTT  M  =  (Q  ,  X,  A,  R,  F)  there  exists  a  GSDTT  F  = 
(G,  A 1 ,  r/  R ' )  such  that  W  and  F  define  the  same  translation. 


Proof:  Given  M=(Q,  I,  A,  R,  F)  we  can  construct  F=(G,  a',  Y ,  R*>  as 


follows:  Let  A*  =  A,  and  G  =  (V,  l',  P,  S)  where  X 

S  =  A  ,  q  is  the  accepting  state  and  P  and  R*  are 
qo 

For  each  (b)  -►  (q,t)  in  R  where  b  z  X q  e  Q;  t  r 
t  to  P,  R'  respectively.  For  each  (b,  ,  ...,  q^) 

e  X  q.  r  Q  t  e  A  (X.  )  add  A  ♦  bLArt  ,  A  ,  ..., 

n  1  *  h  q  qi  q2 

R*  where  t*  is  the  same  as  t  except  that  each  X.  in 


=  X.  V=L A  | q  c  Q>, 
q 

formed  as  follows: 

A*  add  Aq  *  b'  Aq1  * 
♦  (q,t)  where  n>U,  b 

A  J,  A  -►  t  1  to  P  , 
qn  q1 

t  is  replaced  by 
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The  next  thing  is  to  show  that  M  and  F  define  the  same  translat ion: 

If  <b,t>  e  M,  b  e  Z  ,  t  e  A  ,  it  can  be  found  st rai ghtf orwara  from  the 

o  * 

construction  procedure  of  F  that  <b,t>  e  F. 

If  <r,t>  e  M,  the  depth  of  r  is  k  and  if  the  statement  that  <r,,tl> 

e  M  =>  <r',t'>  e  F  holds  for  all  <r,,t'>  where  the  depth  of  r*  is  K-1 , 

then  suppose  r= bCr^  r^,  rp3  where  b  e  Z n,  r-  e  Z*,  the  depth  of  r^ 

_<  K-1.  Since  <r,t>  e  M,  there  exists  a  rule  in  R  of  the  form  (b, 

•  --/  Qn>  *  (qg,  u)  where  is  accepting  state  and  u  e  A^CX^,  besides, 

t  is  equal  to  the  tree  of  u  after  each  X.  in  u  being  replaced  by 

where  u.  e  M(q.,r.),  then  there  exists  a  rule  in  R':A  ♦  u  with  each 

111  qG1 

x.  of  u  being  replaced  by  A  and  a  corresponding  rule  in  P  ;  A  * 

1  n 


bCA  ,  A  ,  . ..,  A  3,  and  A  -►  r.  (*  by  theorem  in  Brainerd  L59j)  the 
q1  q2  qn  qi  1 


depth  of  all  n  <  k,  therefore  by  induction  we  know  that  <r,t>  c  M  => 
<r,t>  e  F.  Similarly,  we  can  show  that  <r,t>  e  F  =>  <r,t>  e  M  therefore 
M  and  F  define  the  same  translation  . 


Theorem  3.2:  For  any  TDTT  M=(Q,  Z,  A,  R,  Q)  there  exists  a  GSDTT  F=(G, 

A*,  r,  R1)  such  that  M  and  F  define  the  same  translation. 

Proof :  We  prove  it  by  constructing  a  GSDTT  F=(G,  A',  r,  R')  as  follows: 

Let  G= (V,  I1,  P,  S),  Z'  =  Z,  a'  =  A,  such  that  L (G)  =  (*  the 

existence  of  G  is  proved  in  Brainerd  C593).  Let  r  =  V  x  Q  i.e.  suppose 

the  number  of  states  in  Q  is  m,  then  A..  ....  A  c  r  for  A  e  V.  For 

i  m 

each  rule  in  R  :(q,b)  -  w,  where  q  e  Q,  b  e  £  ,  w  e  A  (Q  x  X  )  add  to 

n  "  n 

R ' :  •*  u,  where  u  is  the  tree  of  w  after  each  q'X.  being  replaced  by 

B.q/  corresponding  to  each  rule  A  ♦  bCB^,  B 2,  Bn3  in  P.  Next  step 

is  to  prove  that  <r,t>  c  M  <==>  <r,t>  e  F :  If  <r,t>  e  M  and  r=b,  b  e  £ 


L 
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then  there  exists  a  rule  in  R  :  (qo,b)  -►  t  therefore  there  exists  a  rule 

in  P  :  S  -*■  b,  and  a  rule  in  R':  S  +  t  so  <b,t>  z  F.  If  <r,t>  e  M  ana 

q 

Ho 

r=bCr._  r.,  ....  r  ],  b  t  E  then  there  exist  a  rule  in  R  :  (q  ,b)  ♦  t1 
1  2  n  n  o 

where  t*  e  A  (Q  x  X  )  and  the  terminal  part  of  t1  can  be  matched  with  t. 

*  n 

Therefore  there  exists  a  rule  in  P  :  S  bLB^,  B^,  and  a  rule 

in  R1:  S  ♦  u  where  B.  e  V,  u  is  the  result  of  replacing  each  in  t* 

by  B..  .  If  the  statement  that  <r,t>  c  M  =>  <r,t>  e  F  is  true  for  all  r 
of  depth  K-1  then  it's  also  true  for  all  r  of  depth  K.  We  already  know 
that  it's  true  for  K=1  therefore  by  induction  we  know  that  <r,t>  c  M  => 
<r,t>  z  F.  Similarly,  it  can  be  shown  that  <r,t>  c  F  =>  <r,t>  e  M.  So, 
M  and  F  define  the  same  translation. 

Example  _3 *^3 :  Let  I  =  <+,*,y/c>  be  ranked  alphabet,  where  +  ana  •  huve 
rank  of  2,  and  y  and  c  have  rank  0.  We  construct  an  NGSDT  6  which  takes 
the  formal  derivative  with  respect  to  y  of  the  expressions  represented 
by  input  trees  in  where  c  represents  a  constant.  Let  A  = 

,0,y,C,]>.  Let  G=(Ld,I>,I,A,R,Ld>),  where 

R=L (b,+)*C(d,x^)+(d,X2)3, 

<d,#)  +  CC(d,x,| )  •  (I^^D  +  Cd/X^ )  •  (c^x^DD, 

(d,y)-1,(d,c)-0>  U 
t  ( l,a)+o  |  ae*Cy,c»  [J 
LCl,o)-»‘C(I,x^)a(I,X2)j|ac'C+,#>>. 

This  TOTT  can  be  replaced  by  a  GSDTT : 


j 
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(d)  k  denotes  the  sequence  number  for  counting  the  sequence  of 
rule  applications. 

(3)  let  X  =  S,  i=1,  <u,v>  =  <(S,  ,  0,  1),  (S^  ,  0,  1)  >. 

(4)  Select  the  first  rule  of  P  for  X.  j=1. 

(5)  replace  node  (X,  ,  l,  k)  of  u  with  rule  X.  in  P.  If  X  is  X  * 

jo  jo 

bCB^,  B.,,  ...,  then  replace  the  node  with  (X,  b,  j,  i ) L ( ,  (j, 

i),  (Bm,  ,  0,  i)3. 

(6)  Check  comput ibi L i ty  for  u  and  r,  if  compatible  go  to  (?)  else  go  to 

(11). 

(7)  For  each  node  (X^,  ,  0,  k)  of  v,  where  n  is  any  number,  replace  it 

with  rule  X.  in  R.  If  a  node  in  X.  is  a  terminal  b,  it  is  written  as 

jn  jn 

(X  ✓  b,  j,  i),  if  a  node  in  X.  is  a  nonterminal  B  ,  it  is  written  as 
n  j  n  m 

(Bm'  /  0,  i),  go  to  (8). 

(8)  Check  compatibility  for  v  and  t,  if  compatible,  go  to  (9),  else 

(12). 

(9)  Check  if  there  is  any  nonterminal  node  left  in  u,  if  yes,  let  X  be 

the  next  nonterminal,  i=i+1,  go  to  (4),  if  not,  go  to  (10). 

(10)  Check  compatibility  for  u  and  r,  v  and  t,  if  compatible,  find  the 
parse  by  tracing  down  tree  u,  report  success  and  go  to  (12),  if  not,  go 
to  (12). 

(11)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 

(X,  ,  j,  k).  go  to  (13). 

(12)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 

(X,  ,  j,  k),  delete  the  subtrees  being  added  to  v,  restore  their  roots 

with  <Xn,  ,  j,  k)  go  to  (13). 

(13)  check  if  there  are  new  rules  available  for  X,  if  yes,  set  j=j+1. 
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go  to  (5),  if  not  go  to  (14). 

(14)  If  X=S,  then  finished,  exit.  If  not,  then  backtrack,  take  the 
immediate  father  of  node  (X,  ,  j,  k)  as  new  X,  i=i-1,  and  go  to  (4) 
(Figure  3.1  shows  the  flowchart  for  this  algorithm) 

Example  3.4:  The  translation  rules  of  the  GSDTT  defined  in  example  3.3 
is  labeled,  reordered  and  written  as  follows: 


P: 


R: 


<s10>  S  -  y 


<S,1>  S1  '  1 


cs20)  s  -  c 


(S21)  S1  -  0 


(S30>  S  *  + 


/\ 

A  B 


(s31>  s1  *  + 


/\ 

*i  Bi 


<SW>  s 


/\ 

A  B 


<s4i>  si 


/\  A 

A1  B2  A2  B1 


(A1q)  A  -►  y 


(A^  ^  )  A  .  *■  1  (A^  ^2  *  y 


(a20>  A  -  C 


^ A 2 ^ /  A^  *  0  (A22)  ^2  *  ^ 
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(a30)  A  >  +  <A31)  Ai  -  +  (A32)  A,  * 


/\ 

A  B 


/\ 

A,  8, 


/\ 
»2  8, 


(A32}  a2  "*  + 


/\ 
A2  B2 


<A40>  A  "  * 


/\ 

A  B 


(A^^  )  >  + 


(A42)  A2 


A  /\ 

A1  B2  A2  B1 


/\ 

A2  S2 


(B10)  B  *  y 


(B1 1 )  B1  -  1 


(B12)  B2  ♦  y 


cb20)  B  -  C 


<b21)  b1  -  0 


(B22)  B2  "■  c 


(B30>  B  + 


A 

A  B 


(b31>  b1  -  + 


/\ 

A,  B, 


tB32>  B2  '  * 


*2  62 


«W  B  ‘  ‘ 


A 

A  B 


<b*i>  bi  - 


<B«>  B2 


•  • 

A  A 

A,  B2  A2  6, 


/\ 

A2  02 


The  following  diagram  shows  the  analysis  of  the  input  tree  pair 

+  + 

<  /\  ,  /\  >  with  respect  to  this  6SDTT ; 

c  y  0  1 


success ful  analysis 
report  and  backtrack 


A  more  bets’' led  diagram  containing  information  about  each  noae's 


configuration  is  shown  as  follows*. 


<($,  *,  3,  nt(A 


0,  1>> 


3,  DC  (A,  y, 
3,  mu^. 


/  / 

It 


<(S,  t,  3,  1)C(A,  ,  3,  1),  (B,  ,  3,  1)3, 

(S,,  ♦,  3,  1HU,,  ,  3,  1),  (B,,  ,  3,  m> 


// 

1,  2),  CB,  ,  3,  1)3. 

/  3.  1),  (0r  ,  3,  D3> 


<CS,  +  ,  3,  1 )  C  C A,  •,  4, 

4,  2),  (9.  4,  2>j,  (0, 

(Sv  ♦,  3,  1U(AV  ,  3, 
,  3,  1)j> 


2)CU,  , 

,  3/  1)3, 


<(S,  ♦  ,  3,  1H(A,  +  ,  3,  2) C ( A,  , 

3.  2).  (0.  .  3,  2)3.  IB,  ,  3.  1)j, 

<SV  +,  3.  1)C<AV  ,  3,  1).  <8^,  ,  3.  1)3> 


<(S,  +,  3.  1  )CA,  c /  2.  2) ,  (B,  ,  3.  13 

(S.,  ♦,  3.  1)L(A  .  ,  3.  D.  (Bv  ,  3. 


1>J> 


3.1  Flow  chart  for  the  parsing  of  a  GSOTT 
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3 . 3  Stochastic  Genera  l i zed  Syntax-Pi rec  t eg  T  ree  T  rans  tat  ion 

Def  i  ni  t  i  on  5.^8:  A  stochastic  generalized  syntax-directed  tree 

translation  (SGSDTT)  is  a  four-tuple  F  =  (G,  A,  r,  R),  where: 

(1)  G  -  U',  S,N  is  a  regular  tree  g  'rimer; 

(2)  A  is  a  finite  set  of  output  symbols; 

(3)  r  is  a  finite  set  of  distinct  translation  symbols  of  the  form 

t^CA),  where  i  is  an  integer  and  A  is  in  V  -  tS>,  plus  the  symbol  S* . 
whenever  it  is  possible  to  do  so  without  confusion,  we  will  Denote  i • (A) 
by  A^.  .  We  call  k.  the  i-th  translation  symbol  associated  with  A. 

(A)  R  is  a  function  which  associates  with  a  probability  value  p, 

G  <.  p  <  1  each  production  A  *  a  in  P,  and  a  set  of  semantic  rules  = 

A-,  =  ...v  A  =  i  },  in  which  each  6.  is  a  tree  1  n  (r  |J  A)  ,  sucri 

1'  2  2  m  m  '  i  * 

that  all  translation  symbols  appearing  in  6.  are  trensltffon  s/mols 
associated  with  nonterminals  appearing  in  a. 

Definition  3.9:  An  SGSDTT  is: 

(a)  unrestricted  i 4  for  each  rule  p:  A  ♦  a,8  (6  ^ s  the  set  of  semantic 
rules  associated  with  A  -  a)  the  probability  o  is  not  conditioned 
on  other  rules  or  events. 

(b)  proper  if  for  each  nonterminal  A  the  probabi l i t i es  of  all  rules  in 
which  A  is  the  leftside  nonterminal  sum  to  'i . 

We  consider  only  unrestricted,  proper  translation. 

Definition  5. 1C:  The  stochastic  translation  from  i  to  produced  by  F 
with  starting  symbol  S  is  the  set 
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f (F,S> 


<■  Cx/y/p)  |x  in  l y  in  A^,  P 


n  f  (i) 

L  II  Pij<x,y)> 

i=1  j=1 


where  there  are  n  distinct  standard  derivations  (e.g.,  breadth-first)  of 
(x,y)  with  f(i)  steps  in  the  ith  one,  and  P— (x,y)  is  the  probability 
assigned  to  the  jtn  rule  of  the  ith  derivation. 

Similar  to  the  stochastic  syntax  analysis  for  context-free 
programmed  languages  CIO],  depending  on  the  type  of  stochastic  selection 
for  the  next  rule,  we  have  two  distinct  types  of  stochastic  syntax 
analyzer  for  tree  translation.  The  first  type  is  one  with  a  selection 
method  which  searches  for  the  most  likely  rule  first,  while  the  second 
type  is  one  with  a  selection  method  which  randomly  selects  a  rule  for  a 
nonterminal  according  to  the  distribution  over  all  possible  alternatives 
for  the  nonterminal  considered. 

The  stochastic  syntax  analyzer  that  searches  ior  the  most  lueLy 
rule  first  is  a  nondetermini s t i c  syntax  analyzer  in  which  the  rules  are 
arranged  in  descending  order  of  magnitude  of  their  associated 
probabilities  C54],  The  first  rule  for  each  nonterminal  is  the  one  witn 
the  highest  probability,  while  the  last  rule  tor  each  nonterminal  is  tne 
one  correspond! ng  to  the  lowest  probability.  Clearly,  this  is  a 
stochastic  syntax  analyzer  with  a  fixed  strategy  lIuJ.  The  procedure 
for  this  analyzer  is  given  in  Algorithm  3.2.  A  flow  chart  is  shown  in 
Fig.  3.2. 

The  stochastic  syntax  analyzer  with  a  random  strategy  is  formed 
from  the  nondetermini st i c  syntax  analyzer  by  incorpora t i ny  a  stochastic 
selection  algorithm  in  selecting  the  next  rule  when  alternatives  are 
available.  At  each  step  the  conditional  probability  distribution  is 


used  in  selecting  the  next  rule  among  all  available  alternatives  UQj. 
Suppose  that  there  are  3  translation  rules  for  nonterminal  A,  y^,  y ^  and 
with  P^,  ?2,  being  their  respective  probabilities,  and  y ^  has  been 
previously  applied.  After  the  backtracking,  y ^  and  y ^  are  the  only 
available  rules.  Hence,  either  y or  y^  can  be  selected  according  to 

i  • 

their  conditional  probability  distribution  P^2'Y3^Y1^  =  where 

i  i 

?2  ~  P2>/^2+P3')  anci  P3  ~  P3^P2+P3^*  Yhe  Proceciure  ^or  this  analyzer  is 
given  in  Algorithm  3.3  and  a  flow  chart  is  shown  in  Fig.  o.3. 
(Algorithm  3.2  of  the  SGSDTT  parser  with  a  fixed  strategy  is  essentially 
the  same  as  Algorithm  3.1  of  the  nonstochastic  GSDTT  parser.  Algorithm 
3.3  of  the  SGSDTT  parser  with  a  random  strategy  is  almost  the  same  as 
Algorithm  3.1  except  the  rule  selection  method.) 

Algorithm  3.2:  Parsing  for  SGSDTT  with  a  fixeo  strategy. 

Input.  An  SGSDTT  F=(G,  A,  T,  R),  G~(V,  £,  P,  S)  and  an  input  tree  pair 
<r ,t>. 

Output.  A  w  i  the  correct  parses  for  <r,t>.  The  output  "error1'  it  no 
pa  r se  exists. 

Method. 

(1)  For  each  nonterminal  A  in  V,  order  the  alternates  in  R  for  A  in 

descending  order  of  magnitude  of  their  associated  probabilities.  Let 

A  be  the  index  *or  the  ith  alternate  of  A.  Let  A.  be  the  inaex  for 
io  i ) 

the  jth  alternate  in  R  with  respect  to  A.  in  P. 

10 

(2)  Let  <u,v>  be  a  new  tree  pair.  A  4-tuple  (X,  b,  l,  k)  will  be  useo 
to  denote  the  conf igurat ion  of  each  node  in  a  tree: 

(a)  X  denotes  the  nonterminal  of  the  node  before  it  is  rewritten. 
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(b)  b  denotes  the  terminal  of  the  node  after  it  is  rewritten. 

(c)  l  denotes  the  label  of  the  production  rule  for  rewriting  tne 

node. 

Cd)  k  denotes  the  sequence  number  for  counting  the  sequence  of 
rule  applications. 

(3)  Let  X  =  S,  i=1,  <u,v>  =  <(S,  ,  0,  1),  (S  ,  0,  1)  >. 

(4)  Select  the  first  rule  of  P  for  X.  j  =  1. 

(5)  replace  noae  (X,  ,  l,  k)  of  u  with  rule  X.  in  P.  If  X  is  X 

jo  jo 

bCB^  B^,  _ ,  B^],  then  replace  the  node  with  (X,  b,  j,  1)1(6^,  ,  U, 

i),  ...,  (B^,  ,  0,  i)3. 

(6)  Check  compat i bi l i ty  for  u  and  r,  if  compatible  go  to  (7),  else  go 
to  (11). 

(7)  For  each  node  (X^,  ,  0,  k)  of  v,  where  n  is  any  number,  replace  it 

with  rule  X  in  R.  If  a  node  in  X  is  a  terminal  b,  it  is  written  as 

jn  jn 

IX  .  b,  i .  i )  „  if  a  node  in  X  is  a  nonterminal  fa  ,  it  is  written  as 
n'  '  '  jn  nr 

(B^,  ,  0,  i),  go  to  (8). 

(8)  Check  compa t ibi l l t y  for  m  and  t,  if  compatible,  go  to  (V)  else 

(12). 

(9)  Check  if  there  is  any  nonterminal  node  Lett  in  u,  it  yes,  let  X  De 

the  next  nonterminal,  i=i+1,  go  to  (4),  if  not,  go  to  (10). 

(10)  Check  compatibility  for  u  and  r,  v  and  t,  if  compatible,  tina  the 
parse  by  tracing  down  tree  u,  report  success  ana  go  to  (12),  if  not,  go 
to  (12). 

(11)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 

(X,  ,  j,  k).  go  to  (13). 

(12)  delete  the  subtree  just  being  addea  to  u,  restore  its  root  with 
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(X,  ,  j,  k),  delete  the  subtrees  being  added  to  v ,  restore  their  roots 

with  (X^,  ,  j,  k)  go  to  (13). 

(13)  check  if  there  are  new  rules  available  for  X,  if  yes,  set  j=j+1. 
go  to  (5),  if  not  go  to  (14). 

(14)  If  X=S,  then  finished,  exit,  if  not,  then  backtrack,  take  the 
immediate  father  of  node  (X,  ,  j,  k)  as  new  X,  i=i-1,  and  go  to  (4) 
(Figure  3.2  shows  a  flow  chart  for  this  algorithm). 

Algorithm  3.3:  Parsing  for  SGSDTT  with  a  random  strategy. 

Input.  An  SGSDTT  F=(G,  A,  T,  R),  G=(V,  I,  P,  S)  and  an  input  tree  pair 
<r,t>. 

Output.  All  the  correct  parses  for  <r,t>.  The  output  "error"  if  no 
parse  exists. 

Method . 

(1)  For  each  nont  errr  ina  l  A  in  V,  order  the  alternates  in  R  tor  A .  Let 

A.  be  the  index  for  the  ith  alternate  of  A.  Let  A  De  the  "index  for 

10  i  j 

the  ]th  alternate  in  R  with  respect  to  A.^  in  P. 

(2)  Let  <u,v>  be  a  new  tree  pair.  A  4-tuDle  (X,  o,  l,  k)  will  be  useo 
to  denote  the  conf i gurat ion  o*  each  node  in  a  tree: 

(a)  X  denotes  the  nonterminal  of  the  node  before  it  is  rewritten. 

(b)  b  denotes  the  terminal  of  the  node  after  it  is  rewritten. 

(c)  i  is  a  register  denoting  which  rules  have  been  applied  for  the 
node  before. 

(d)  k  denotes  the  sequence  number  for  counting  the  sequence  of 
rule  applications. 

(3)  let  X  =  S,  i=1,  <u,v>  =  <(S,  ,  0,  1),  (S^  ,  0,  1)  >. 

(4)  Select  the  first  rule  of  P  for  X  statistically,  set  j  =  the  index 


s: 


of  the  rule. 

(5)  replace  node  (X,  ,  l,  k)  of  u  with  rule  X.  in  P.  If  X  is  X  ^ 

jo  jo 

bCB^,  B^,  ...,  B^],  then  replace  the  node  with  (X,  b,  l' ,  i)L (b^,  ,  0, 

i),  ...,  (B  ,  ,  0,  i)3,  l'  =  Q  OR  y) ,  where  y  =  2J ,  the  jth  bit  of  the 
binary  form  of  y  is  1,  showing  that  the  jth  rule  of  X  is  now  being 
applied. 

(6)  Check  compatiL  lity  for  u  and  r,  if  compatible  go  to  (7)  else  go  to 
(11). 

(7)  For  each  node  (X^,  ,  Q,  k)  of  V,  where  n  is  any  number,  replace  it 

with  rule  X.  in  R.  If  a  node  in  X  is  a  terminal  b,  it  is  written  as 

jn  jn 

(X  ,  b,  i),  if  a  node  in  X.  is  a  nonterminal  B  ,  it  is  written  as 
(B^,  ,  0,  i),  go  to  (8). 

(8)  Check  compatibility  for  v  and  t,  if  compatible,  go  to  (9)  else 

(12). 

(9)  Check  if  there  is  any  nonterminal  nude  left  in  u,  if  yes,  let  X  De 

the  next  nonterminal,  i=i+1,  go  to  (4),  if  not,  go  to  (10). 

(10)  Check  compat ibi li ty  for  u  and  r,  v  and  t,  if  compatible,  find  the 
parse  by  tracing  down  tree  u,  report  success  and  go  to  (12),  if  not,  go 
to  (12). 

(11)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 

(X,  ,  j,  k).  go  to  (13). 

(12)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 

(X,  ,  j,  k),  delete  the  subtrees  being  added  to  v,  restore  their  roots 

with  (X^,  ,  j,  k)  go  to  (13). 


(13)  check  if  there  are  new  rules  available  for  X,  if  yes,  select  next 
rule  for  X  statistical ly,  set  j  =  the  index  of  the  rule,  go  to  (3),  if 
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not  go  to  (14). 

(14)  If  X=S,  then  finished,  exit.  If  not,  then  backtrack,  take  tfte 
immediate  father  of  node  (X,  ,  j,  k)  as  new  X,  i=i-1,  and  go  to  (4) 
(Figure  3.3  shows  a  flow  chart  for  this  algorithm). 


For  each  nonterminal  A,  arrange  the  alternates  of  A 
in  descending  order  of  the  magnitude  of  associated 
probab 11 i ty . 
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CHAPTER  4 

AN  APPLICATION  OF  TREE  TRANSLATION  TO  TRAFFIC  IMAGE  SEQUENCE  ANALYSIS 

_4.J_  Introduction 

This  chapter  gives  an  illustrative  example  of  applying  the  tree 
translation  model  described  in  Chapter  3  to  the  design  of  an  automatic 
traffic  image  sequence  analysis  system-  Traffic  image  is  a  popular 
subject  of  study  in  time-varying  image  analysis.  The  input  of  the 
analysis  system  is  assumed  to  be  a  sequence  of  images  of  a  traffic 
intersection  scene.  An  example  is  shown  in  Figure  4.1.  In  this  system, 
each  input  image  is  divided  into  a  set  of  fixed-size,  f i xed-pos^ t i on 
windows.  Each  image  is  then  represented  as  a  tree  of  wb7cb  each  node 
corresponds  to  a  specific  window  in  the  image.  Each  node  is  labeled  to 
indicate  the  occurrence  or  nonoccurrence  of  a  vehicle  in  its 
corresponding  window.  A  tree  translation  scheme  is  then  used  to 
describe  the  motion  of  the  vehicles  in  the  image  sequences.  Matching  of 
vehicles  in  different  images  is  performed  in  the  form  of  a  tree 
translation  parsing. 


Scene  Representation 
4.2.1  Image  representat i on 

Each  image  is  first  divided  into  a  set  of  windows.  An  appropriate 


window  size  is  selected  so  that  each  window  contains  at  most  one 


vehicle.  In  general,  the  window  length  selected  should  not  be  greater 
than  the  length  of  the  smallest  vehicle.  Window  size  could  be  set  at  a 
very  small  value.  However,  a  smaller  window  size  will  result  in  a  more 
complex  translation  scheme.  There  are  many  ways  to  divide  the  image. 
An  example  is  shown  in  Figure  4.2.  Only  a  part  of  the  road  area  is 
considered.  A  tree  represents! ion  for  the  image  is  also  shown  in  Figure 
4.2.  Each  window  corresponds  to  a  node  in  the  tree  representat ion 
regardless  there  is  a  vehicle  in  this  window  or  not.  Each  vehicle  is 
considered  as  only  a  point.  This  point  could  be  the  centroid,  or  any 
easi ly-recogni 2able  corner-point  of  the  vehicle.  But  after  this 
reference  point  is  selected,  this  point  should  be  used  consistently  for 
each  vehicle  in  the  whole  image  sequence.  The  centroid  of  a  vehicle 
(considering  only  the  x-y  plane)  is  selected  in  this  system.  A 
vehicle's  position  is  represented  by  the  location  of  its  centroid.  Even 
though  one  large  vehicle  could  occupy  more  than  one  window,  only  the 
window  where  this  vehicle's  centroid  resides  is  considered  as  containing 
this  vehicle.  If  no  vehicle  occurs  in  a  window,  then  the  corresponding 
node  of  this  window  is  labeled  by  'O'.  If  there  is  a  vehicle  in  a 
window,  then  the  corresponding  node  is  labeled  by  the  quantization  value 
of  this  vehicle's  orientation  (see  Figure  4.3).  Additional  information 
(includes  vehicle  size,  actual  centroid  position,  orientation  angle)  is 
attached  to  the  node  for  other  purposes  (e.g.  speed  calculation, 
clearing  of  ambiguity  occurring  in  tree  translation  parsing).  The 
determination  of  a  vehicle's  orientation  and  centroid  position  will  be 
discussed  later.  An  ambiguity  of  tree  translation  parsing  means  that 
different  time-varying  activities  between  images  result  in  the  same 
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Figure  4. 
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2  Divided  intersection  area  and  its  tree  representation. 

(The  root  of  the  tree  is  represented  by  an  encircled  node.) 
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Figure  4.3  Orientation  primitives 
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sequence  of  trees. 


4.2.2  Motion  representat ion 

Tree  translation  rules  are  used  to  represent  vehicle  movement. 
Only  the  movement  from  one  window  to  another  is  described.  The  movement 
within  the  same  window  is  not  considered.  But  the  information  about 
vehicle’s  centroid  position  and  orientation  is  attached  to  the  node. 
This  information  can  be  used  if  necessary.  It  is  assumed  that  each 
vehicle  can  move  no  more  than  the  length  of  the  smallest  vehicle  between 
two  consecutive  images. 

Example  4.1 :  Figure  4.4a  shows  an  intersection  area  being  divided  into  4 
windows.  One  vehicle  is  coming  in  from  the  south  while  another  one  is 
moving  toward  the  south.  The  following  rules  are  required  for  this 
movement : 

S  -  7  0 

/\  /\ 

A  B  ,  A  B 

A-0,7  C  *  0  ,  3 

where  0  means  no  vehicle  existence  and  i  means  the  existence  of  a 
vehicle  in  direction  i,  1  i  _<  8. 

Example  4.2;  Figure  4.4b  shows  that  one  vehicle  is  moving  from  the  north 
to  the  west  and  two  others  are  moving  from  the  south  to  the  north.  Both 
the  (i-l)th  and  the  ith  images  have  the  same  intersection  content. 
There  are  two  interpretations  for  the  vehicle  moving  west.  Either  the 


B  +  0  0 

C  ,  C 


k 
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(b) 


Figure  4.4  Intersection  area  contents.  ("♦"  sign  indicates  the 
existence  of  a  vehicle  in  that  direction.) 
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same  vehicle  exists  in  the  same  window  for  both  images  or  the  one  in  the 
(i-l)th  image  has  left  the  intersection  area  and  a  new  one  is  appearing 
in  the  ith  image.  A  similar  interpretation  also  holds  for  those  two 
vehicles  moving  north.  There  can  be  four  different  interpretations  for 
the  variation  between  the  (i-l)th  and  the  ith  images.  But  only  one  set 
of  translation  rules  is  needed  to  describe  this  image  pair. 

S  -  6  6 

/\  /\ 

A  B  ,  A  B 

A  +  0  ,  0 

From  the  assumption  about  vehicle  speed,  this  problem  is  solved  by  using 
vehicle  position  and  orientation  values  attached  to  each  node  of  the 
tree.  For  the  vehicle  with  label  6,  if  the  orientation  value  of  the 
vehicle  in  the  ith  image  is  greater  than  that  of  the  vehicle  in  the 
(i-l)th  image,  then  it  indicates  that  the  vehicle  in  the  (i-l)th  image 
has  left  the  intersection  and  the  vehicle  in  the  ith  image  appears  for 
the  first  time  and  traffic  flow  (the  number  of  passing  vehicles)  is 
increased  by  one.  Otherwise  it  indicates  that  the  vehicles  appearing  in 
the  (i-l)th  and  the  ith  images  are  the  same  ones.  (Orientation  value  is 
counted  counterclockwise)  For  those  vehicles  moving  north,  the  y 
coordinate  of  the  front  vehicle  is  used.  If  the  y  coordinate  of  the 
front  vehicle  in  the  ith  image  is  smaller  than  that  of  the  front  vehicle 
in  the  (i-l)th  image,  then  it  indicates  that  the  front  vehicle  in  the 
(i-l)th  image  has  left  the  intersect  ion,  the  front  vehicle  in  the  ith 


B  *  3  3 

C  ,  C 
C  -  3  ,  3 


92 


Figure  4. 


5  Conditions  of  the  (i-l)th  and  ith  images  for  which  the  use 
of  attribute  value  is  required.  (Each  one  of  (a?  (b)  (c) 
represents  one  of  four  similar  cases.  Blank  windows  could 
contain  vehicle  or  not.  The  attribute  values  of  node 
representing  lower-right  window  will  be  used  for  case  la) 
and  (b).  For  case  (c)  the  attribute  values  of  the  top- 
right  window  will  be  used.) 
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image  was  the  rear  vehicle  in  the  (i-l)th  image,  the  rear  vehicle  in  the 
ith  image  appears  for  the  first  time  and  the  traffic  flow  is  increased 
by  one-  Otherwise  it  indicates  that  the  front  vehicles  in  both  the  ith 
and  the  (i-l)th  images  are  the  same  one  and  so  are  the  rear  vehicles. 
There  are  also  some  other  similar  cases  which  require  the  use  of 
attribute  values  of  each  node  to  resolve  such  an  ambiguity.  Their 
corresponding  translation  rules  are  specified  for  the  requirement  of  the 
comparison  between  node  attribute  values  of  2  input  trees.  Figure  4.5 
shows  the  various  cases  in  which  a  comparison  of  node  attribute  value  is 
requi red. 


4_. 3^  Scene  Ana lys i s 
4.3.1  Feature  extraction 

Due  to  the  similar  property  in  feature  extraction  between  this 
experiment  and  You  and  Fu's  shape  recognition  experiment  C13J,  the 
algorithms  for  boundary  following  and  boundary  smoothing  described  in 
C 1 33  are  applied  for  feature  extraction. 

The  first  step  in  feature  extraction  is  threshold  selection. 
Because  of  the  flat  black  paint  on  the  models,  the  vehicles  look 
uniformly  dark  and  should  create  a  peak  in  the  high  gray  level  region  of 
the  histogram.  The  light  background  is  supposedly  uniform  too  and 
create  a  peak  in  the  low  gray  level  region  of  the  histogram.  A  typical 
histogram  of  the  experiment  is  shown  in  Figure  4.6.  The  peak  at  gray 
level  k^  is  caused  by  the  vehicles.  The  peak  at  gray  level  is  caused 
by  the  background.  The  selected  threshold  is  t. 

Algorithm  Threshold  Selection 
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Input:  A  digital  picture. 

Output:  A  threshold  t. 

Method: 

(1)  Compute  the  histogram. 

(2)  Find  the  peak  of  the  highest  gray  Level  (which  usually 
corresponds  to  the  object).  Let  =  the  gray  level. 

(3)  Find  the  second  peak  of  the  histogram  besides  the  one  found  in 
(2).  Let  k^  =  the  corresponding  gray  Level. 

(4)  Find  the  lowest  valley  between  the  above  two  peaks  and  let  t  = 
gray  level  corresponding  to  the  valley. 

(5)  Terminate. 

After  a  threshold  is  found,  the  boundary  for  each  vehicle  can  be 

traced  out.  The  boundary  is  defined  as  a  connection  of  edges  between 

the  object  and  the  background.  The  boundary  is  coded  by  unit  vectors 

with  horizontal  and  vertical  directions.  Each  boundary  is  traced  out  by 

the  following  boundary  following  algorithm  which  is  Led  by  the  contents 

of  a  2x2  window.  Figure  4.7  shows  the  four  possible  configurations. 

The  pixels  A,  B,  C,  and  0  are  defined  relative  to  the  boundary  vector  P. 

The  object  is  to  the  right  of  P,  so  that  A  is  darker  than  the  threshold 

t-  The  background  is  to  the  left  of  P,  so  that  C  is  lighter  than  t.  In 

the  following  algorithm,  u  and  u  are  the  unit  movements,  or  unit 

x  y 

vectors,  in  the  X  and  Y  directions  respectively.  A,  B,  C,  £>  denote  the 
coordinates  of  the  pixels.  G(B)  is  the  gray  level  of  the  pixel 
indicated  by  B.  F  is  the  first  pixel  of  an  object  detected  by  scanning 
the  image. 

Algorithm  4.2:  Boundary  Following  (You  and  Fu  C133) 

Input:  F,  u  ,  u  ,  and  threshold  t. 
x  y 

Output:  A  boundary  chain  U  of  i  unit  vectors. 
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Figure  4.6  A  typical  histogram  of  the  experiment 


Figure  4.7  Four  possible  configurations  of  boundary  following  window 
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Method: 

(1)  Set  P  =  u  ,  i  =  1,  UC1 )  =  P 

A  =  F,  C  =  F-u  ,  S  =  C,  go  to  C3) 

(2)  If  (S  =  C)  then  terminate 
otherwise  i  =  i+1,  UCi)  =  P 

(3)  D  =  C+P 

If  (G(D)  <  t)  then  go  to  (4) 
otherwise  P  =  C-A,  A  =  D,  go  to  (2) 

(4)  B  =  A+P 

If  (G(B)  <  t)  then  go  to  (5) 
otherwise  A  =  B,  C  =  D,  go  to  (2) 

(5)  C  =  B,  P  =  B-D  go  to  (2) 

The  output  from  the  above  boundary  following  algorithm  is  a  string 
of  unit  vectors.  Due  to  the  digitization  grid,  there  are  many  zig¬ 
zag's.  A  smoothing  method  (You  and  Fu  C133)  is  applied  to  transform  a 
string  of  unit  vectors  into  a  string  of  longer  vectors.  This  method  is 
defined  in  the  form  of  an  attributed  finite  transducer. 

In  the  following  definition,  q j 1 s  are  attributed  states-  Each 
state  qj  represents  a  subchain  sj  which  is  accepted  by  not  translated, 
and  which  is  described  by  the  associated  attributes.  £v  denotes  a 
series  of  unit  vector  v,  or  l  times  v.  -v  is  the  negative  of  v,  i.e., 
-v  and  v  have  the  same  length  but  opposite  directions.  A  A  B  denotes 
that  B  follows  A.  6  is  a  mapping  from  Q  x  I,  under  condition  C,  to 
finite  subsets  of  Q  x  0*-  The  mapping  performs  when  condition  C  is 


t  rue. 


For  each  state  transition,  there  is  a  set  of  attribute  rules. 
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The  attributes  of  a  state  may  be  unit  vectors  or  numbers.  In  the 
transition  rule,  i,  m,  n,  v,  u  are  unit  vectors  and  p,  q,  fc,  k  are 
numbers.  The  attribute  conditions  for  a  transition  are  described  above 
the  right  arrow.  For  each  transition,  the  input  unit  vector  is  compared 
with  the  vector  attributes  and  the  attribute  conditions  are  checked. 
Then  the  machine  goes  to  the  next  state  with  the  appropriate  output  and 
transfers  the  attributes  according  to  the  attribute  rules.  Each 
expression  of  the  output  is  a  vector  e  0.  An  example  of  the  smoothing 
effect  of  this  transducer  is  shown  in  Figure  4.8. 

Def ini tion  Attributed  Finite  Transducer  A 

A  is  a  6-tuple,  (Q,  I,  0,  6,  S,  F). 

I  =  the  input  set  consisting  of  4  unit  vectors  and  an  end  marker  $, 

{(1,0),<0,-1),(-1,0),(Q,1),$> 

0  =  the  output  set,  ^ (n^ ,n^)  |  (n..  =  0,n^_^  *  0)  or 

(ru  =  ±1,  n^_..  =  any  integer),  i  =  1,2> 

Q  =  a  set  of  states  with  attributes,  {qj|j  =  0, — ,9> 

q0:  sO  =  X,  empty 

ql  :  si  has  one  unit  vector  v 

v 

q2fc  •  s2  =  — *  ,  i  >  1 

'  iv 

v 

q3v  u:  s3  =  v  A  u,  1  u  (or  J) 


99 


Av 


PA  :  s4  =  Av  A  u,  - \  u  (or  t  ),  a  >  1 

X#  j  V  j  u  ^ 


q5v  u:  s5  =  v  A  u  A  -v,  u  (or  tD  ) 


q6  : 
v,u 


s6  =  v  A  u  A  v,  ^(or  j-) 


Av 


q7A,v,u,k:s7  =  A  u  A  kv. 


(or  i — ) 


kv 


Q8 


v,  A,u‘ 


s8  =  v  A  lu,  | _  (orl  *),A  >  1 


Au 


Av 


q9 


A,v,uk  * 


$9  =  Av  A  u  A  kv  A  “U , 


I _ J 

kv 

A  >  1,  k  >  1 


(or 


S  =  the  initial  state  qO 
F  =  a  set  of  final  states,  <q0> 


6:  (qO,i)  *►  (c.1^,4)),  v  «-  i,  i  e  I— C$> 


(q1m/m)  *  (q2A,v,4))'  v  *  m'  *  *  2 


*  (q3v,u,4,)/  v  *  m'  u  +  j 
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(q6m  n/n^  *  (q0,in+n  /  m+n) 


(q6m,n/in) 


',q7J./v/u,k/4’) '  v 


l,  k  - 

m,  u  «- 


v  ♦  m/ 

(q 7  ,m)  ♦  ( q 7  ,.<+>). 

P/m,n,q  £,v,u,k'9  '  u  «•  n. 


Cq7 


p/m,n,q 


,n) 


p>2q 

+  (q1v,(p-1)m/  2qm+n),  v  ♦  n 


q/2<p<2q 

**■  (ql  ,  (p+q)m+n) ,  v  ♦  n 


p<q/2  4  *  q“P 

-  <q4t  ,  2pm+n),  v  *  m 

u  «-  n 


(q7p,m,nrq'  n) 


(q9 n  ,  ,  <|>) 
A/v/u/k/  v 


A  «-  p 
k  «-  q 
m 

u  ♦  n 


<q8  -n) 

m/q  /n 


(q8 


v,*,u 


v  ♦  m 
,  <$>),  u  ♦  n 
t  «■  q+1 


(q8 


m,q  ,n 


,m) 


2 

n 

l  +  p 
k  «-  q+1 
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q<4 


q-1 


(q4„  ,m+n)  v  <-  n 


q>4 


l  ♦  q-2 

(q4l  v  u'm+2n)  v  *  n 
u  +  m 


Cq8 


m,q^n 


,-m) 


q<4 


Jl  ♦  q-1 

(q4n  ,  m+n)  v  «-  n 
u  «*  -m 


q>4 


n  ♦  q-2 


(q4 


*,u,v' 


m+2n)  v  ♦  n 
u  +  -m 


(q9p,m,n,q'm)  *  (q2*,v'  ♦>  v 


m+n+1 

m 


1-2 


(q9, 


p,m,n,q'-n)  *  (q2l,v'  (p+q)m+n)'  v  «.  -n 


Cq9 


^  m  „  .-m)  +  (q3  -  (p+q)m+n), 

p,m/n,q'  M  v,u'  K  H  '  u  «-  -i 


v  +  -n 
m 


(q0/$)  >  (q0,4>) 


(q1m/$)  -  (q0,m) 
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<q2  ,$>  Cq0,pm) 

P 

<q3m/n'P  <q0'm+n) 

<q4n  m  n't*  *  (q0'P m+n) 

P/uVn 

*  O'0-"' 

tq6m,n'*)  *  <q0/2nl+n) 

(q7p,m,n,q'^  +  W,  (p+qhn+n) 

(q8m,q,n'$)  *  (q0'm+qn) 

(q9  ,$)  ♦  (q0,(p+q)m) 

p,m,n,q/,p  ' 

4.3,2  Shape  analysis 

Consider  the  top  view  first.  An  attributed  shape  grammar  is 
adapted  for  vehicle  shape  representation  Cl 3 J .  To  find  the  orientation 
and  the  centroid  position  of  a  vehicle,  special  symbols  are  marked  in 
the  production  rules  of  the  shape  grammar  to  specify  which  pair  of  angle 
points  can  be  used  to  find  the  orientation  and  which  pair  of  angle 

points  can  be  used  to  find  the  centroid  position.  Suppose  that  a 

* 

derivation  of  a  shape  contour  is  S  *  F.A-  ...  F.A.  ...  FA.  ...  A  , 

I!  j  j  m 

where  F's  are  curve  primitives  and  A’s  are  angle  primitives,  and  it  is 
known  in  the  grammar  inference  stage  that  the  direction  of  the  line 
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segment  extending  from  the  point  of  angle  A.  to  the  point  of  angle 

decides  the  orientation  of  this  vehicle  and  it  is  also  known  from 

parsing  that  this  derivation  generates  the  input  boundary  string  ... 

V  ...  V  ...  V  with  substring  V  V  ..  ...  V  matching  F  •  .  ...  F . . 
p  q  n  p  P+1  q  i+i  j 

Then  the  position  of  angle  A..  (A ^ )  is  the  same  as  the  position  of  the 
breaking  point  between  vectors  V  .  and  V  (V  and  V  Therefore  the 

orientation  of  this  vehicle  is  decided  with  the  knowledge  of  x  and  y 

coordinates  of  these  two  breaking  points  which  is  obtained  in  the 
feature  extraction  stage.  The  centroid  position  is  calculated 

similarly.  Suppose  that  the  midpoint  of  the  line  segment  connecting  the 
angles  A^  and  A^  is  the  centroid.  Then  the  centroid  position  is  decided 
from  the  x  and  y  coordinates  of  the  angle  points  of  A^  and  A ^ .  You  and 
Fu's  LI  3D  PEE  Earley*$  parser  is  used  for  vehicle  shape  recognition. 

For  example,  consider  a  vehicle  shape  shown  in  Figure  4.9.  The 

shape  is  represented  as  S  *  F^ A-| F2A2F3A3F4A4 *  The  m^dP°^nt  the  L^ne 

segment  connecting  the  angles  A^  and  A^  is  the  centroid.  The  direction 

of  the  line  segment  connecting  the  angles  A^  and  determines  the 

orientation  of  the  vehicle.  After  knowing  that  A^  is  located  at 

(x^y^,  A^  is  located  at  (x-^y^)  and  is  located  at  then  the 

centroid  position  is  found  to  be  ((x^+x^)/2,  (y^+y^J/Z)  and  the 

orientation  of  the  vehicle  has  the  slope  Cy^— y ^ ) / (x^-x^ ) .  The  question 

about  whether  the  actual  direction  should  be  from  A^  to  A^  or  from  A^  to 

will  be  discussed  shortly.  Strictly  speaking,  for  a  simple  shape 

like  the  top  view  of  a  vehicle,  the  shape  analysis  method  described 

above  is  not  really  necessary.  For  instance,  an  easy  way  to  calculate 

the  centroid  Cx  ,y  )  of  a  vehicle  is  to  apply  the  formula:  x  =Ex  /n, 
cr  c  c  1 


carting  Point 


Figure  4. 


9  An  example  of  vehicle  shape 
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y  =J]y./n,  where  the  summation  is  taken  over  all  points  on  the  boundary 
i 

and  n  is  the  total  points  on  the  boundary.  The  orientation  can  also  be 
easily  determined  after  searching  for  the  right  angles  around  the 
boundary  and  finding  out  the  longest  line  segment  between  these  angles. 
But  compared  with  this  simple  method,  the  method  described  earlier  in 
this  subsection  does  serve  as  a  way  for  the  determination  of  the 
centroid  and  orientation  of  a  general  shape. 

Due  to  the  symmetrical  property  of  the  top  view  of  a  vehicle,  a 
vehicle  in  any  direction  could  have  two  possible  opposite  orientations. 
For  example,  the  shape  of  a  vehicle  moving  northwest  could  also  be 
evaluated  as  southeast  -  oriented.  This  problem  is  solved  as  follows: 
Consider  a  vehicle  located  in  the  lower-right  window  of  the  intersection 
area.  If  its  shape  is  detected  as  horizontally  oriented  (either  to  the 
east  or  west),  then  it  is  considered  as  moving  east  with  orientation  1. 
Similarly,  if  its  shape  is  detected  as  vertically  oriented,  then  it  is 
considered  as  moving  north  with  orientation  3.  If  its  shape  is 
northwest  -  oriented  (in  other  terms,  southeast  -  oriented),  it  could 
have  orientation  4  or  8.  this  ambiguity  is  resolved  from  the  knowledge 
of  the  last  image  and  the  assumption  about  vehicle  movement  limitation 
between  consecutive  images.  Specifically,  if  there  exists  a  vehicle 
with  orientation  8  in  either  the  lower-left  or  the  lower-right  window  of 
the  intersection  area  in  the  (i-l)tb  image,  then  this  northwest-oriented 
vehicle  in  the  lower-right  window  of  the  ith  image  should  have 
orientation  8  instead  of  4.  Otherwise  this  vehicle  should  have 
orientation  4.  For  a  northeast-oriented  vehicle  in  the  lower-right 
window,  there  is  only  one  choice:  orientation  2.  The  way  of 
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determining  the  orientation  of  vehicles  in  the  lower-right  window 
described  above  also  applies  to  other  windows. 

When  an  input  image  does  not  represent  the  top  view,  the  vehicle 
shape  variation  and  occlusion  problems  happen.  For  shape  variation  and 
simple  occlusion,  the  problem  can  still  be  solved  with  the  help  of  a 
generalized  error-correct ing  PEE  parser  (6ECPEEP)  LI  3D .  But  when  the 
number  of  vehicles  appearing  in  the'scene  increases,  a  serious  occlusion 
would  make  the  extraction  of  vehicle  centroid  position  and  orientation 
information  very  difficult.  Therefore,  in  order  to  make  the  monitor 
working  even  under  heavy  traffic  conditions,  it  would  be  better  to  use 
the  top-view  of  the  scene  as  the  input. 

4.3.3  Tree  translation  parsing 

After  the  tree  representation  for  the  ith  image  is  obtained,  it  is 
parsed  by  the  tree  translation  parser  along  with  the  tree  representation 
for  the  (i-l)th  image.  (Parsing  process  starts  from  the  second  image.) 
The  translation  parser  is  similar  to  the  one  shown  in  Chapter  3.  But  as 
mentioned  before,  some  conf igurat ions  of  input  tree  pair  still  require 
the  use  of  attributes  attached  to  the  node  to  decide  the  actual  activity 
between  two  consecutive  images.  After  the  parsing,  the  parser  either 
reports  the  output  or  asks  for  a  comparison  of  attribute  values.  (The 
conditions  requiring  the  use  of  attributes  and  which  attributes  to  be 
used  have  been  discussed  earlier  and  illustrated  in  Figure  4.5.) 

Inference  of  _a  Structure-Preserved  T ree  Translation  Schema 

The  tree  translation  involved  here  is  structure-preserved  (in  other 
words,  substitution  t ransf ormat ion  only).  The  inference  of  the  tree 
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translation  schema  is  therefore  essentially  a  tree  grammar  inference 
problem:  Let  £  be  the  input  alphabet,  A  be  the  output  alphabet  of  a  tree 
translation  and  S+  =  be  the  tree  translation 
sample  set.  x..  and  y^  have  the  same  tree  structure.  Form  a  new 
alphabet  £  =  £x  a.  Transform  each  tree  pair  <x-,y^>  into  a  single 
tree  over  £  by  combining  the  label  of  each  node  of  x.  and  the  label  of 


its  corresponding  node  of  y^ .  The  tree  translation  sample  set  S+  is  now 

transformed  into  a  set  of  tree  samples.  Apply  the  conventional  tree 

grammar  inference  procedure  (Moayer  and  Fu  C61 J )  to  infer  the  tree 

grammar  which  can  generate  these  tree  samples.  Then  transform  this  tree 

grammar  into  the  required  tree  translation  schema  by  converting  each 

ab 

production  rule  into  a  tree  translation  rule.  Ce.g.  if  A  ^  is  a 

rule  of  the  tree  grammar,  where  a  e  £  and  b  e  a,  then  the  converteQ  tree 

a  b 

translation  rule  is  A  ,  /l.\  )*  The  whole  procedure  is 


given  as  follows: 


81  Bn  B1  8n 


Algorithm  4.3:  Inference  of  a  st ructured-preserved  tree  translation 
schema 

Input:  A  tree  translation  sample  set  S+  =  {<x^ ,y^>,  x2'y2># * ’<xn'yn>^/ 

where  x.  and  y.  have  the  same  tree  structure  (tree  domain), 
i  i  ' 

input  alphabet  £  and  output  alphabet  A. 

Output:  a  tree  translation  schema  which  generates  S+. 

Method:  1.  Let  £  =  £  x  A,  S+*  = 

2.  For  each  <x.,y^>  e  $+  f 

add  a  tree  (a:  D  ♦  £  >  to  S*+, 

where  D  is  the  domain  of  x.  and  y., 

i  'i' 

a(b)  =  (b)a^ .  (b)  for  b  e  0, 
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a  .  and  o  .  are  the  tree  function  for  x.  and  y.  respectively. 

*  •  /  *  11 

3.  Infer  a  tree  grammar  for  S++  (Moayer  and  Fu  C61 □ ) : 

Step  1:  Selection  of  proper  substructures,  here  the  aim  is  to 
select  components  which  have  high-repetitive 
occurrence . 

Step  2:  Infer  subgrammar  G_.  for  each  substructure 

Step  3:  Let  be  the  union  of  the  subgrammars  G..  inferred  in 
step  2.  Relabel  the  variables  and  eliminate  redundant 
production  rules  *0  eliminate  ambiguity  and  erroneous 
generation  paths. 


4.  Form  a  tree  translation  schema  T ( £,a,y,R,S)  by  adding  one 
a  b  ab 

rule  A  -*■  /. \  ,  X.\  to  R  for  each  rule  A  X..\  of  the 


Bi  B„  Bi  B, 

1  .n  .  .1  ^  n 

grammar  generated  in  3. 


81  8n 


Consider  the  tree  translation  involved  in  the  traffic  scene  as  an 
example.  The  input  alphabet  and  output  alphabet  are 
£  =  ,A  =  <0,1 ,2, 3, 4, 5,6,7, 8>.  To  reduce  the  number  of  samples, 
consider  only  the  image  pairs  in  which  no  vehicle  exists  in  the 
intersection  area  of  the  first  image  and  one  vehicle  comes  into  the 
intersection  area  of  the  second  image.  The  tree  translation  sample  set 
contains : 


0— G  0—0 


0  0  0  2 


0—0  0—0 

<  i  i,i  i  > , 

0  0  0  3 
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0—0  0—0  0—0  0— A 

<  I  1,1  l>,<  I  1,1  l>, 

0004  0000 


0—0  0—5  0—0  0—6 

<  I  1,1  I  >  ,  <  I  1,1  I  >  , 
0000  0000 


0—0  6—0  0—0  7—0 

<  I  1,1  l>,<  I  1,1  I  >  , 

0000  0003 


0—0  8—0  0—0  0—0 

<  I  1,1  I  >  ,  <  I  1,1  I  >  , 
0000  0080 


0—0  0—0  0—0  0—0 
<  I  1,1  I  >  ,  <  I  1,1  I  > . 
0  0  1  0  0  0  2  0 


These  tree  translation  samples  are  transformed  into  trees  over  £  x  4. 


Then  a 

tree  grammar 

is 

generated 

from  these  tree 

samples.  For  example. 

0—0 

0—0 

00—00 

<  1  1 

,  1  1  > 

becomes 

1  I- 

The  tree  sample 

set  contains: 

0  0 

0  2 

00  02 

00—00 

00—00 

00—00 

1  1  , 

I  1  ✓ 

1 

1 

✓ 

00  02 

00  03 

00 

04 

00—04 

00—05 

00- 

-06 

1  1  , 

1  1  / 

1 

1 

/ 

00  00 

00  00 

00 

00 

Ill 
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00 


00 


00 


04 


8‘*i 


1 


05 


06 


00 


*7*  ! 


A1  -  00  A2  -  08 


A3  -  01  A4  -  02 


C1  -  03  C2  -  04 


where  S,  A^/...,A4,  B^,. . . and  C2  are  nonterminals. 

The  resulting  tree  translation  schema  contains  the  following 
translation  rules: 


0  0 
S  -  /\  ,  /\ 

A1  B1  A1  B1 


0  0 
/\  *  /\ 
A1  B2  A1  B2 


1 

L 


\ 

a 
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A1  >  0  ,  0  A^  -  0  ,  8 


A3  -  0  ,  1  A4^0,2 

C,j  +  0  ,  3  C£  ^  0  ,  4 

Since  there  are  4  windows  in  the  intersection  area,  there  are  4 
nodes  in  each  tree  and  8  nodes  in  each  translation  pair.  There  are  9 

different  possible  labels  for  each  node:  TQ,1 ,2, . . . ,8>.  Theoret i ca l  ly, 
4  8 

there  are  9  different  trees  and  9  different  translation  pairs.  But 

0—0  0—0 

after  deleting  some  unrealistic  cases  (e.g.,  a  tree  like  |  |  or  |  |  is 

0—0  0—0  0  5  0  7 

not  reasonable,  a  translation  pair  like  <  |  |  ,  |  |  >  is  not  possible 

0  3  0  0 

due  to  the  assumption  about  speed  limitation),  there  are  25  different 
tree  patterns  and  186  different  translation  pairs  left.  These  186 

translation  pairs  are  divided  into  2  groups.  No  vehicle  moves  out  of 
the  intersection  area  for  each  translation  pair  of  group  I,  while  one  or 
more  vehicles  have  moved  out  of  the  intersection  area  for  each 

translation  pair  of  group  II.  Group  I  consists  of  70  members.  The  12 
translation  pairs  given  in  the  above  inference  example  are  part  of  these 
70  pairs.  Group  II  consists  of  116  members.  These  116  translation 
pairs  are  further  classified  into  12  subclasses  according  to  the 
specific  traffic  type.  These  12  subclasses  are  EW,  ES,  EN,  WE,  WS,  WN, 
SE,  SW,  SN,  NE,  NW  and  NS,  where  E,  W,  S  and  N  denotes  east,  west,  south 
and  north  respectively  and  EW  represents  the  subclass  of  translation 
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pairs  for  which  one  vehicle  has  moved  from  the  east  to  the  west  ana  left 
the  intersection  area,  ES  represents  the  subclass  of  translation  pairs 


for  which  one  vehicle  has  moved  from  the  east  to  the  south,  etc.  For 
0—3  0—0  0—4  0—0 

example,  <  |  I  ,  |  I  >  belongs  to  the  subclass  SN  and  <  |  I  ,  I  I  > 
0000  0000 

belongs  to  the  subclass  EN.  There  are  some  translation  pairs  which 

0—3  0—0 

belong  to  more  than  one  subclass.  For  example,  <  |  |  ,  |  |  >  belongs 

7  0  0  0 


to  both  subclasses  SN  and  NS. 


Implementation 

The  whole  experiment  of  traffic  image  sequence  analysis  was  set  up 
in  the  laboratory  of  pattern  processing  and  advanced  automation  at 
Purdue  University.  The  set-up  for  data  collection  is  shown  in  Figure 
4.1.b.  A  white  background  with  black  vehicles  is  used  for  the  traffic 
scene.  Road  areas  and  non-road  area  were  specified.  The  digital 
picture  sequence  was  taken  directly  through  a  TV  scanner  which  is 
located  above  the  traffic  intersection.  The  digitization  process  was 
controlled  interactively  through  a  PDP  11/45  computer  in  the  laboratory. 
Before  di gi t i zat ion,  we  adjustea  the  relative  distance  and  the  focus  of 
the  TV  scanner  to  obtain  a  reasonably  clear  picture  on  a  TV  monitor. 
Then  everything  kept  stationary  except  the  vehicles  during  the 
digitization  of  the  whole  image  sequence.  After  each  digitization  of  a 
picture,  the  vehicles  were  moved  to  new  locations  for  the  next  picture. 

The  traffic  analysis  system  using  the  proposed  tree  translation 
parsing  is  implemented  in  Fortran  IV  under  Unix  system  on  a  P DP11/45 
computer.  The  average  time  required  for  the  analysis  of  one  image  is 
about  15  seconds  (the  actual  computer  time  for  each  individual  image 
depends  on  the  number  of  vehicles  in  the  image).  About  90%  of  the  time 
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is  spent  on  the  feature  extraction  and  the  shape  analysis  for  tree  node 
Labeling. 

Figure  A. 11  shows  the  image  contents  of  some  of  the  image  sequence. 
Figure  4.12  shows  their  corresponding  tree  representat ions  around  the 

intersection  area.  The  correspondence  between  the  nodes  and  the 

windowed  regions  is  shown  in  Figure  4.2.  The  8  nodes  corresponding  to 
the  windows  surrounding  the  intersection  area  are  included  to  indicate 
where  the  vehicles  come  from  and  where  they  go.  Figure  4.13  shows  the 
traffic  information  extracted  from  the  image  sequence.  This  experiment 
illustrates  an  example  of  the  application  of  tree  translation  models  to 
the  analysis  of  time-varying  patterns.  In  chapter  2,  we  let  denote 

the  set  of  pattern  representat ions  of  some  object  occurring  at  time 
denote  the  set  of  pattern  represenations  occurring  at  time  t^  and  use 

the  concept  of  translation  to  model  the  relation  between  and  lor 

the  time-varying  process  of  the  pattern  representation  between  time  t^ 
and  time  t^).  In  this  traffic  scene  experiment,  the  "object"  under 
study  is  the  content  of  the  intersection  region  in  a  traffic  scene. 
and  are  a  set  of  trees  a:  D  +  X,  where  D  is  the  tree  domain 

<0,0.1 ,0.2, 0.2. 1>  and  L  is  the  primitive  set  <0,1 ,2,3, . . .,8>.  Actually 
and  are  the  same  set.  As  mentioned  in  section  4.4,  (or  L^)  has 
25  patterns  and  there  are  186  possible  translation  pairs  between  and 
<2*  These  186  translation  pairs  are  divided  into  2  groups.  Group  I 
involves  no  traffic  flow  out  of  the  intersection  area.  Group  II 
involves  traffic  flow  and  is  further  classified  into  12  subclasses.  At 
each  time  instant  t=t^,  the  tree  representations  of  the  (i-l)th  and  i-th 


images  are  parsed  to  decide  the  membership.  If  they  belong  to  group  I 
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then  there  is  no  traffic  flow  out  of  the  intersection  area  and  no 

information  update  is  required.  If  they  belong  to  group  II,  then  an 

action  for  the  specific  subclass  is  taken:  update  the  traffic 

information  for  the  specific  path.  For  example,  the  tree 

representations  of  the  intersection  area  for  image  SC01  and  image  SC02 

0—5  5—0 

of  Figure  A. 11  are  |  |  and  |  |  which  are  generated  by  the  following 

0  0  10 

trans lat ion  ru les : 


0  5  5  0 

SI  -  /  \  ,  /  \  B12  -  I  ,  I 

A3  B12  A3  B12  C3  C3 


A3  -0,1  C3  -»  0  ,  0 


They  belong  to  group  I.  Therefore  no  action  is  taken  and  the  traffic 

information  is  unchanged.  At  the  next  time  instant,  the  tree 

5—0 

representations  for  image  SC02  and  image  SC13  of  Figure  4.10  are  |  | 

0 — 0  1  0 

and  |  |  ,  which  are  generated  by  the  following  translation  rules: 

0  1 

5  0  0  0 

S2  ♦  /  \  ,  /  \  B1?  ♦  |  ,  | 

A12  B17  A12  B17  C13  C13 


A12  -  1  ,  0  C13  -  0  ,  1 


They  belong  to  the  subclass  EW  of  group  II.  Therefore  the  traffic 
information  is  updated  and  the  number  of  vehicles  moving  from  the  east 
to  the  west  is  incremented  by  one.  (In  case  that  a  translation  pair 
belongs  to  two  subclasses  as  mentioned  in  section  4.4,  then  it  is 
required  to  update  the  traffic  information  of  the  paths  specified  for 
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both  subclasses.)  The  image  sequence  of  this  experiment  does  not  have 
any  significance  of  physical  meaning.  It  is  only  used  to  show  how  the 
analysis  system  is  applied.  (Since  there  are  186  possible  translation 
pairs  (in  other  words,  186  time-varying  patterns  between  consecutive 
images),  in  order  to  go  through  all  of  the  time-varying  patterns,  the 
required  number  of  images  is  far  greater  than  186.  It  is  similar  to  the 
input  sequence  required  to  test  every  state  transition  of  a  finite-state 
machine.)  This  experiment  shows  a  form  of  pattern  recognition  system 
under  time-varying  situation.  It  also  demonstrates  one  way  to  extract 
information  from  an  image  sequence  through  a  series  of  pattern 
translation  operations.  On  the  other  hand,  although  the  testing  image 
sequence  is  successfully  analyzed,  further  improvement  of  the  system  is 
still  required.  Similar  to  the  cases  of  Chow  and  Aggarwal  [43]  and 
Aggarwal  and  Duda  C4 2D,  this  system  assumes  a  fairly  simple  image 
condition  for  preprocessing.  Therefore,  to  make  the  system  working 
under  noisy  conditions,  more  sophisticated  feature  extraction  technique 
will  certainly  be  required. 

4_.j6  Cone lusions  and  Discussions 

While  many  traffic  scene-related  research  activities  [23,68,69,80] 
concentrate  on  segmentation  techniques,  the  proposed  traffic  analysis 
system  emphasizes  the  representation  of  vehicle  motion  and  assumes  that 
there  is  little  difficulty  in  segmentation,  which  is  the  main  assumption 
of  the  system.  The  advantages  of  the  proposed  system  include:  (1)  each 
moving  object  (vehicle)  is  allowed  to  have  movement  ranging  from  0  to 
the  length  of  the  smallest  vehicle  between  consecutive  images,  (2)  the 
matching  process  is  performed  through  a  tree  translation  parsing  which 
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is  very  efficient  in  processing,  (3)  there  is  no  need  to  do 
sophisticated  prediction  using  information  of  the  past  history,  (4) 


there  is  no 

need 

to 

keep 

the  past 

information  except  trie  tree 

representation 

of 

the 

Last 

image,  and 

(5)  the  ability  to  describe  an 

image  scene  and  to  model  an  image  sequence.  The  experiment  conducted 
here  is  a  2-lane  traffic.  When  the  number  of  traffic  lanes  increases, 
the  required  analysis  system  is  essentially  the  same  except  that  more 
windows  are  required  and  the  corresponding  tree  representation  contains 
more  nodes.  Figure  4.14  shows  a  4-lane  example.  There  are  16  windows 
in  the  intersection  area.  The  scene  is  represented  as  a  tree  with  16 


nodes. 


Figure  4.11  Image  contents  of  the  first  8  images  in  the  testing  image 
sequence 
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SC01 

0  0 

0—0— 5—0 

1—0  0—0 

0  3 

SC  02 

0  0 


0—5— 0—0 


0—1  0—0 


0  3 

SC03 

0  0 


5—0— 0—0 


0—0  1—0 


0  3 

SC04 

0  0 


0—0— 0—5 


0—0  0—1 


0  3 


SC05 

0  0 


0—0— 5—0 


0—0  0—0 


0  3 

SC06 

0  0 


0—5— 0—0 


1—0  0—0 


0  3 

SC07 

0  0 


5—0— 0—0 

I  I 

0—1  0—0 

0  3 

SC08 

0  0 

0—0— 0—0 

0—0  1—0 

0  3 


Figure  4.12  Tree  representations  for  the  images  of 
Figure  4.11. 
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Figure  4.13  Traffic  information  extracted  from 
the  image  sequence 
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Figure  4.14  A  4-Lane  intersection  area  being  divided  into  16  windows 
and  its  tree  representation  (with  the  encircled  node  as  the 
root) 


CHAPTER  5 


CONCLUSIONS  AND  SUGGESTIONS  FOR  FURTHER  RESEARCH 
1*1  Cone  lusions 

A  syntactic  method  for  the  analysis  of  time-varying  image  patterns 
is  proposed  and  studied.  This  method  utilizes  a  translation  schema  to 
model  time-varying  properties.  A  syntactic  deformation  model  is  first 
applied  to  transform  the  i-th  image  into  the  (i+1)-th  image  of  an  image 
sequence.  Then  the  concept  of  translation  in  formal  language  theory  is 
used  as  a  mechanism  to  characterize  the  dynamic  process  of  the  sequence. 
In  order  to  analyze  high-dimensional  patterns,  the  string  translation 
schema  is  extended  to  that  for  trees.  Formulation  of  stochastic 
translation  is  also  presented  for  the  modeling  of  stochastic  properties 
of  time-varying  patterns. 

A  traffic  scene  analysis  problem  is  analyzed  using  the  proposed 
method.  Each  input  image  is  presented  as  a  tree  structure.  Tree 
translation  is  used  to  model  the  variation  of  image  content  between 
consecutive  images.  A  parsing  algorithm  for  the  tree  translation  is 
applied  to  match  vehicles  in  each  pair  of  consecutive  images.  The 
advantages  of  this  system  are:  (1)  Each  moving  object  is  allowed  to  have 
larger  flexible  movement  between  consecutive  images,  (2)  the  matching 
process  is  performed  through  a  tree  translation  parsing  which  is  very 
efficient  in  processing,  (3)  there  is  no  need  to  do  sophisticated 
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prediction  using  information  of  the  past  history,  (4)  there  is  no  need 
to  store  the  past  information  except  the  last  image,  and  (5)  has  the 
potential  for  simulation  of  the  time-varying  process.  This  system 
assumes  little  difficulty  in  the  segmentation  of  moving  object,  which  is 
its  main  limitation. 


5.2  Suggestions  for  Further  Research 

The  problem  of  time-varying  pattern  analysis  is  a  new  research 
topic  and  is  getting  increasing  attention.  Although  the  results  from 
the  proposed  syntactic  method  appear  to  be  quite  sat isfactory,  there  are 
still  some  topics  requiring  further  investigations. 

(1)  The  translation  models  studied  in  this  report  include  string 
translation  and  tree  translation  models.  There  are  some  other  patterns 
which  are  better  being  represented  as  graphs  CIO, 96, 98, 99] .  Recently 
Jacobus,  Chien  and  Selander  analyze  the  similarities  in  the  structure  of 
abstract  forms  produced  from  stereoscopic  motion  picture  sequences  using 
a  graph  matching  technique  C72].  A  pair  of  images  contains  the  scenes 
of  a  3-D  object  before  and  after  the  movement.  The  informtion 
concerning  the  boundary  and  surface  of  the  object  at  each  image  is 
represented  as  a  graph.  A  graph  matching  method  is  used  to  match  the 
corresponding  parts  of  the  two  graphs  (assuming  small  overall  movement). 
In  order  to  make  graph  representat ion  applicable  for  general  time- 
varying  patterns,  it  would  be  desirable  to  study  graph  language 
translation  and  stochastic  graph  translation. 

(2)  You  and  Fu  C13D  study  the  syntactic  shape  recognition  using 
attributed  grammars  with  promising  results.  Tsai  and  Fu  C14,96J  report 
an  attributed  pattern  deformation  model  and  find  that  sometimes  it  is 
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more  powerful  tc  use  attributed  pr  mitives  than  discrete  primitives.  In 
order  to  take  attributes  into  consideration  in  time-varying  pattern 
problems,  one  way  is  to  extend  the  translation  models  to  translation 
models  with  attributes:  attributed  string  translation,  attributed  tree 
translation  and  attributed  graph  translation, 

(3)  Tree  translation  is  applied  to  analyze  traffic  image  sequence 
in  Chapter  4.  In  order  to  further  understand  the  usefulness  of  tree 
translation,  it  would  be  desirable  to  study  traffic  scenes  using  tree 
translation  through  an  extensive  simulation  experiment. 

(4)  The  problem  of  cardiac  motion  analysis  is  getting  more 
attentions  [91,92,93,94,953.  The  use  of  computerized  tomography 
technology  is  also  becoming  more  popular.  It  would  be  interesting  to 
study  the  applicability  of  the  translation  models  described  in  this 
research  to  the  problem  of  heart  wall  motion  representation,  or  more 
generally,  to  the  problem  of  representat ion  of  data  extracted  from 
computerized  tomography. 


128 


LIST  Of  REFERENCES 


[ID  K.  S-  Fu,  Sequential  Methods  in  Pattern  Recognition  and  Machine 
Learning.  Academic  Press,  New  York,  1968. 

[2D  K.  Fukunaga,  Introduction  to  Statistical  Pattern  Recogni t ion. 
Academic  Press,  New  York,  1972. 

[3D  E.  A.  Patrick,  Fundamentals  of  Pattern  Recognition.  Prentice- 
Hall,  Englewood  Cliffs,  New  Jersey,  1972. 

[4D  H.  C.  Andrews,  Int roduction  to  Mathemat i ca l  Techniques  in  Pattern 
Recognition,  Wiley,  New  York,  1972. 

CSD  R.  0.  Duda  and  P.  E.  Hart,  Pattern  Classification  and  Scene 
Analysis.  Wiley,  New  York,  1973. 

[63  C.  H.  Chen,  Statistical  Pattern  Recognition.  Hayden,  Washington, 
D.C.,  1973. 

[7D  K.  S.  Fu  and  P.  H.  Swain,  On  Syntactic  Pattern  Recognition,  In 
Software  Engineering  (J .  T.  Tou,  ea.),  Vol.  2,  Academic  Press, 
New  York,  1971. 

C8D  W.  F.  Miller  and  A.  C.  Shaw,  Linguistic  Methods  in  Picture 
Processing-a  Survey.  Proc.  AFIPS  Fall  Joint  Comput.  Conf.,  San 
Francisco,  1968,  pp.  279-290. 

[9D  Pattern  Recognition  3  (1971);4  (1972).  Special  issues  on 
syntactic  pattern  recognition. 

[10D  K.  S.  Fu,  Syntactic  Methods  in  Pattern  Recognition.  New  York, 
Academic  Press,  1974. 

[113  S.  Y.  Lu  and  K.  S.  Fu,  Stochastic  error-correcting  syntax 
analysis  for  recognition  of  noisy  patterns.  IEEE  Tjrans.  on 
Computers ,  Vol.  026,  No.  12,  Dec.  1977. 

C12D  S.  Y.  Lu  and  K.  S.  Fu,  Error-correct ing  tree  automata  for 


syntactic 
027,  Nov. 

pattern 

1978. 

recogni t ion. 

IEEE 

Trans,  on 

Computers,  Vol. 

K.  C.  You 

and  K. 

S.  Fu,  Syntactic 

shape  recognition  using 

attributed 

University, 

grammar 

.  Technical 

Report  TR-EE 

78-38,  Purdue 

129 


C 1 43  W.  H.  Tsai  and  K.  $.  Fu,  A  pattern  deformation  model  and  bayes 
error-correct ing  recognition  system.  Technical  Report  TR-EE 
78-26,  Purdue  University. 

Cl 5 3  A.  A.  Arking  and  R.  C.  Lo  and  A.  Rosenfela,  An  evaluation  of 
fourier  transform  techniques  for  cloud  motion  estimation. 
Computer  Science  Technique  Report  TR-351 ,  University  of  Maryland, 
Jan.  1975. 

1 

C16J  Abstracts  of  the  workshop  on  computer  analysis  of  time-varying 
i magery.  Apri l  1979. 

Cl  73  H.  Nagel,  Analysis  techniques  for  image  sequences,  1978 
Internat ional  Joint  Conference  on  Pattern  Recognition. 

Cl 83  J.  A.  Leese,  C.  S.  Novak  and  V.  R.  Taylor,  The  deterrmnat ion  of 
cloud  pattern  motions  from  geosynchronous  satellite  image  data. 
Pattern  Recognition,  2,  1970,  279-292. 

Cl 93  R.  P.  Futrelle  and  G.  C.  Speckert,  Extraction  of  motion  data  oy 
interactive  image  processing,  1978  Proc.  of  Pattern  Recognition 
and  Image  Processing  Conference,  405-408,  Chicago,  Illinois. 

C20T  W.  N.  Martin  and  J.  K.  Aggarwal,  Survey:  Dynamic  scene  analysis. 
Computer  Graphics  Image  Processing,  7,  1978,  356-374. 

C21 3  L.  F.  Hubert  and  L.  F.  Whitney,  Wind  estimation  from 
geostationary  satellite  pictures.  Monthly  Weather  Review,  99, 
1971,  665-672. 

C22J  K.  Wolferts,  Special  problems  in  interactive  image  processing  for 
traffic  analysis.  1974  International  Joint  Conference  on  Pattern 
Recognition,  1-2. 

C230  M.  Onoe  and  K.  Ohba,  Digital  image  analysis  of  traffic  flow. 
1976  International  Joint  Conference  on  Pattern  Recognition, 
803-808. 

C243  R.  T.  Chien  and  V.  C.  Jones,  Acquisition  of  moving  objects  and 
hand-eye  coordination.  1975  Internat ional  Joint  Conference  on 
Artificial  Intelligence,  737-741. 

C25D  R.  Jain  and  H.  H.  Nagel,  On  a  motion  analysis  process  for  image 
sequences  from  real  world  scenes.  IEEE  workshop  on  pattern 
recognition  and  artificial  intelligence,  Princeton,  N.J.,  1978. 

C263  C.  A.  Rosen  and  D.  Nitzan,  Use  of  sensors  in  programmable 
automation.  Computer,  10,  Dec.  1977,  12-23. 

C273  B.  Neumann,  Interpretat ion  of  imperfect  object  contours.  1978 
International  Joint  Conference  on  Pattern  Recognition. 


1 


130 


[283  T.  Uno,  M.  Ejiri  and  T.  Kogunaga,  A  method  of  real-time 
recognition  of  moving  objects  and  its  application.  Pattern 
Recogni t ion,  8,  (1976),  201-208. 

[293  R.  Eskenazi  and  R.  T.  Cunningham,  Real-time  tracking  of  moving 
objects  in  TV  images.  IEEE  workshop  on  Pattern  recognition  and 
artificial  intelligence,  Princeton,  N.J.,  1978. 

[303  C.  K.  Chow  and  T.  Kaneko,  Automatic  boundary  detection  of  the 
left  ventricle  from  cineangiograms .  Computers  and  Bimedical 
Research  5  (1972)  388-410. 

[313  C.  K.  Chow,  S.  K.  Hilal  and  K.  E.  Nigbuhr,  X-ray  image 
subtraction  by  digital  means.  IBM  J.  Res.  Develop.  17  (1973) 
206-218. 

C323  K.  Hachimura,  M.  Kuwahara  and  M.  Knoshita,  Left  ventricular 
contour  extraction  from  radioisotope  angi ocardiograms  and 
c lassi f i cat i on  of  left  ventricular  wall  motion.  1978 
International  Joint  Conference  on  Pattern  Recognition. 

C333  S.  A.  Johnson  et  al..  Bioimage  synthesis  and  analysis  from  x-ray, 
gamma,  optical  and  ultrasound  energy.  Digital  Processing  of 
Biomedi ca l  Images,  (K.  Preston  and  M.  Onoe  eds.)  1976,  2U3-226. 

C343  G.  T.  Herman  and  H.  K.  Liu,  Dynamic  boundary  surface  detection. 
Computer  Graphics  Image  Processing  7,  1978,  130-138. 

C353  H.  K.  Liu,  Two-  and  three-dimensional  boundary  detection. 
Computer  Graphics  Image  Processing  6,  1977,  123-134. 

C363  P.  H.  Heintzen  et  al.  Automated  video-angiocardiographi c  image 
analysis.  Computer  8,  1975,  55-64. 

[373  M.  Yachida,  M.  Asada  and  S.  Tsuji,  Automatic  motion  analysis 
system  of  moving  objects  from  the  records  of  natural  processes. 
1978  International  Joint  Conference  on  Pattern  Recognition. 

C383  D.  Davenport  et  al.  The  investigation  of  the  behavior  of 
microorganisms  by  computerized  television.  IEEE  Trans,  on 
Biomedical  Eng.  8ME-17  (1970)  230-237. 

[393  Y.  Ariki,  T.  Kanade  and  T.  Sakai,  An  interactive  image  modeling 
and  tracing  system.  1978  Internat iona l  Joint  Conference  on 
Pattern  Recognition. 

[403  M.  Takagi  and  K.  Sakaue,  The  analysis  of  moving  granules  in  a 
pancreatic  cell  by  digital  moving  image  processing.  1978 
International  Joint  Conference  on  Pattern  Recognition. 


r 


131 


[413  R.  C.  Lo,  and  J.  A.  Parikh,  A  study  of  the  application  of  Fourier 
transforms  to  cloud  movement  estimation  from  satellite 
photographs.  Computer  Science  Technical  Report  TR-242,  University 
of  Maryland,  1973. 


[42D 

J.  K.  Aggarwal  and 

R.  0.  Duda, 

Computer 

analysis 

of 

moving 

polygonal  images, 

966-976. 

IEEE  Trans. 

Computers 

C-24, 

Oct . 

1975, 

C433 

W.  K.  Chow  and  J. 

K.  Aggarwal, 

Computer 

analysis 

of 

planar 

curvilinear  moving 
179-185. 

images,  IEEE 

Trans. 

Computers 

C-26 

,  1977, 

[443  J.  L.  Potter,  Scene  segmentation  by  velocity  measurements 
obtained  with  a  cross-shaped  template,  4IJCAI,  pp.  803-808, 
Tbilisi,  Georgia,  U.S.S.R.,  Sept.  1975. 

[453  R.  Nevatia,  Depth  measurement  by  motion  stereo.  Computer  Graphics 
Image  Processing,  5,  1976,  203-214. 

[463  M.  S.  Ulstad,  An  algorithm  for  estimating  small  scale  differences 
between  two  digital  images.  Pattern  Recognition,  5,  1973, 

323-333. 

C473  R.  L.  Lillestrand,  Techniques  for  change  detection,  IEEE  Trans. 
Computers  C-21,  1972,  654-659. 

[483  G.  G.  Pieroni,  A  method  for  analyzing  dynamic  processes 
represented  by  chains  of  maps. 

[493  Y.  Ariki,  A  system  for  analyzing  time-varying  image  patterns, 

[503  0.  T.  vonRamm,  F.  L.  Thurstone,  Cardiac  imaging  using  a  phased 

array  ultrasound  system.  I.  System  Design.  Circulation.  Vol. 
53,  No.  2,  pp.  258-261,  1976. 

[513  A.  V.  Aho  and  J.  D.  Ullman,  The  Theory  of  Parsing,  Translat ion, 
and  Compiling,  Vol.  1,  Englewood  Cliffs,  N.J.:  Prentice-Hall, 
1972. 

[523  L.  W.  Fung  and  K.  S.  Fu,  "Stochastic  syntactic  decoding  tor 

pattern  classif ication,"  IEEE  Trans,  on  Comput.,  Vol.  C-24,  No. 
6,  July  1975. 

[533  D.  J.  Rosenkrantz,  programmed  grammars — a  new  device  for 

generating  formal  languages,  Ph.D.  dissertation,  Columbia  Un  v.. 
New  York  (1967). 

[543  P.  H.  Swain  and  K.  S.  Fu,  "Stochastic  programmed  grammars  tor 

syntactic  pattern  recognition,"  Pattern  Recognition,  Vol.  4, 
1972. 


132 


[553  K.  S.  Fu  and  B.  K.  Bhargava,  "Tree  systems  for  syntactic  pattern 
recognition,"  IEEE  Trans,  on  Comput.,  Vol.  022,  No.  12,  Dec. 
1973. 

[563  S.  W.  Zucker,  "Toward  a  model  of  texture,"  Computer  Graphics  and 
Image  Processing,  5,  1976,  pp.  190-202. 

[573  A.  V.  Aho  and  J.  D.  UUman,  Translations  on  a  context-free 
grammar.  Inform.  Cont r.  19,  1971,  439-475. 

[583  B.  Baker,  Generalized  syntax  directed  translation,  tree 
transducers,  and  linear  space,  SIAM  J.  on  Computing,  Vol.  7,  No. 
3,  Aug.  1978. 

[593  W.  Brainerd,  Tree  generating  regular  systems.  Information  and 
Control,  14,  1969,  pp.  217-231. 

[603  H.  C.  Lee  and  K.-S.  Fu,  "A  stochastic  syntax  analysis  procedure 
and  its  application  to  pattern  classification,"  IEEE  Trans. 
Comput .,  Vol.  C— 21 ,  July  1972,  pp.  660-666. 

[613  8.  Moayer  and  K.  S.  Fu,  "A  tree  system  approach  for  fingerpnng 
pattern  recognition,"  IEEE  Trans,  on  Comput.,  Vol.  C-25,  No.  3, 
March  1976. 

[623  S.  Y.  Lu  and  K.  S.  Fu,  A  syntactic  approach  to  texture  analysis. 
Computer  Graphics  and  Image  Processing  7,  1978,  303-330. 

[633  S.  Y.  Lu  and  K.  S.  Fu,  Stochastic  tree  grammar  inference  for 
texture  synthesis  on  discrimination,  1978  Proc.  of  IEEE  Pattern 
Recognition  and  Image  Processing  Conference,  Chicago. 

[643  R.  C.  Lo,  J.  Mohr  and  J.  A.  Parikh,  Applications  of  Fourier 
transform  methods  of  cloud  movement  estimation  to  simulated  and 
satellite  photographs.  Computer  Science  Technical  Report  TR-292, 
University  of  Maryland,  1974. 

[653  R.  C.  Lo  and  J.  Johr,  Application  of  enhancement  and  thresholding 
techniques  to  Fourier  transform  cloud  motion  estimates.  Computer 
Science  Technical  Report  TR-326,  University  of  Marlyland,  1974. 

[663  E.  A.  Smith  and  D.  R.  Phillips,  Automated  cloud  tracking  using 
precisely  aligned  digital  ATS  pictures,  IEEE  Trans.  Computers, 
C— 21 ,  1972,  715-729. 

[673  J.  0.  Limb  and  J.  A.  Murphy,  Estimating  the  velocity  of  moving 
images  in  television  signals.  Computer  Graphics  and  Image 
Processing,  4,  1975,  311-329. 


133 


[683  H.  H.  Nagel,  Formation  of  an  object  concept  by  analysis  of 
systematic  time  variations  in  the  optically  perceptible 
environment.  Computer  Graphics  and  Image  Processing,  7,  1978, 
149*194. 

C69]  R.  Jain  and  H.  H.  Nagel,  On  the  analysis  of  accumulative 
difference  pictures  from  image  sequences  of  real  world  scenes, 
IEEE  Trans.  Pattern  Analysis  Machine  Intelligence,  Vol.  PAMI-1 
Apr.  1979,  206*213. 

[703  R.  Jain,  D.  Militzer,  and  H.  H.  Nagel,  Separating  nonstationary 
from  stationary  scene  components  in  a  sequence  of  real  world  TV 
images,  1977  Proc.  IJCA1,  Cambridge,  MA,  Aug.  1977. 

[713  S.  Ullman,  The  Interpretation  of  Visual  Motion,  Cambridge,  MA, 
MIT  Press,  T979. 

[723  C.  J.  Jacobus,  R.  T.  Chien  and  J.  M.  Selander,  Motion  detection 
and  analysis  by  matching  graphs  of  intermediate-level  primitives, 
IEEE  Trans.  Pattern  Analysis  and  Machine  Intelligence,  Vol. 
PAMI-2,  No.  6,  Nov.  1980. 

[733  J.  O'Rourke  and  N.  I.  Badler,  Model-based  image  analysis  of  human 
motion  using  constraint  propagation,  IEEE  Iran*,.  Pattern  Analysis 
and  Machine  Intelligence,  Vol.  PAMI-2,  No.  6,  Nov.  1^80. 

[743  J.  K.  Tsotsos,  etc.,  A  framework  for  visual  motion  understanding, 
IEEE  Trans.  Pattern  Analysis  and  Machine  Intelligence,  Vol. 
PAMI-2,  No.  6,  Nov.  1980. 

[753  J.  W.  Roach  and  J.  K.  Aggarwal,  Determining  the  movement  of 

objects  from  a  sequence  of  images,  IEEE  Trans.  Pattern  Analysis 
and  Machine  Intelligence,  Vol.  PAMI-2,  No.  6,  Nov.  1980. 

[763  M.  Yachida,  M.  Ikeda  and  S.  Tsuji,  A  plan-guided  analysis  of 

cineangiograms  for  measurement  of  dynamic  behavior  of  heart  wall, 
IEEE  Trans.  Pattern  Analysis  and  Machine  Intelligence,  Vol. 
PAMI-2,  No.  6,  Nov.  1980. 

C773  M.  Yachida,  M.  Asada,  and  S.  Tsuji,  Automatic  analysis  of  moving 
objects  from  the  records  of  natural  processes. 

[783  S.  Tsuji,  M.  Asada,  and  m.  Yachida,  Tracking  and  segmentation  of 
moving  objects  in  dynamic  line  images,  IEEE  Trans.  Pattern 
Analysis  and  Machine  Intelligence,  Vol.  PAMI-2,  No.  6,  Nov.  1980. 

[793  J.  W.  Roach  and  J.  K.  Aggarwal,  Computer  tracking  of  objects 

moving  in  space,  IEEE  Trans.  Pattern  Analysis  and  Machine 

Intelligence,  Vol.  PAMI-1,  No.  2,  AprTTT?79. 


134 


[80]  W.  B.  Thompson,  Combining  motion  and  contrast  for  segmentation. 
IEEE  Trans.  Pattern  Analysis  and  Machine  Intelligence,  Vol. 
PAMI-2,  No.  6,  Nov.  1980. 

[81]  C.  L.  Fennema  and  W.  B.  Thompson,  Velocity  determination  in 
scenes  containing  several  moving  objects.  Computer  Graphics  and 
Image  Processing,  9,  1979,  301-315. 

[823  A.  Z.  Meiri,  On  monocular  perception  of  3-D  moving  objects,  IEEE 
Trans.  Pattern  Analysis  and  Machine  Intelligence,  Vol.  PAMI-2, 
No.  6,  Nov.  1980. 

[833  F.  J.  Maryanski  and  M.  G.  Thomason,  Properties  of  stochastic 
syntax-directed  translation  schemata,  Int  l .  J_.  of  Computer  and 
Information  Science,  Vol.  5,  No.  2,  1979. 

[843  M.  G.  Thomason,  Stochastic  syntax-directed  translation  schemata 
for  correction  of  errors  in  context-free  languages,  IEEE  Trans. 
Computer,  C-24,  1975. 

[853  K.  S.  Fu  and  T.  L.  Booth,  Grammatical  inference:  introduction  and 
survey  -  Part  I  and  II,  IEEE  Trans.  Sys .  Man  Cybernetics,  Vol. 
SMC-5,  No.  1  &  4,  1975. 

[863  J.  M.  Brayer  and  K.  S.  Fu,  A  note  on  the  k-tail  method  of  tree 
grammar  inference,  IEEE  Trans.  System  Man  and  Cybernetics,  April 
1977. 

[873  Barry  Levine,  Derivatives  of  tree  sets  with  applications  to 
grammatical  inference. 

[883  S.  W.  Zucker,  R.  A.  Hummel  and  A.  Rosenfeld,  An  application  of 
relaxation  labeling  to  line  and  curve  enhancement,  IEEE  Trans. 
Comput .,  Vol.  C-26,  Apr.  1977. 

[89]  W.  Scacchi,  Visual  motion  perception  by  intelligent  systems, 
Proc .  Pattern  Recog.  Image  Proc.,  Aug.  1979. 

C90]  J.  B.  Garrison,  et  al..  Quantifying  regional  wall  motion  and 
thickening  in  two-dimensional  echocardiography  with  a  computer- 
aided  contouring  system,  1977  Proc.  Computers  in  Cardiology. 

[913  P.  D.  Clayton,  et  al.,  A  computer  system  for  the  cardiovascular 
laboratory,  1974  Proc .  Computers  in  Cardiology. 

[923  B.  R.  Hieb,  et  al.,  A  computerized  system  for  segmental  analysis 
of  sequential  left  ventricular  ci neangiogram  frames,  1976  Proc. 
Computers  in  Cardiology. 

[933  P.  H.  Heintzen,  et  al..  Automated  video-angiocardiographi csh 
image  analysis,  1974  Proc .  Computers  in  Cardiology. 


135 


[94]  R.  8alocchi,  et  al.,  A  system  for  off-line  analysis  of  Left 
ventricular  angiographic  images  by  the  use  of  a  minicomputer, 
1977  Proc .  Computer  in  Cardiology. 

[95]  R.  W.  Brower  and  G.  T.  Meester,  Computer  based  methods  for 
quantifying  regional-  left  ventricular  wall  motion  from 
cineventriculogram^,  1976  Proc.  Computers  in  Cardiology. 

[96]  W.  H.  Tsai  and  K.  S.  Fu,  Error-correcting  isomophisms  of 
attributed  relational  graphs  for  pattern  analysis,  IEEE  Trans. 
System,  Man  and  Cybernetics,  Vol.  SMC-9,  No.  12,  Dec.  1979. 

[97]  T.  I.  Fan  and  K.  S.  Fu,  A  syntactic  approach  to  time-varying 
image  analysis.  Computer  Graphics  Image  Processing,  11,  1979. 

[98]  R.  M.  Haralick  and  J.  S.  Kartus,  Arrangements,  homomorphi sms  and 
discrete  relaxations,  IEEE  Trans.  System,  Man  and  Cybernetics, 
Vol.  SMC-8,  Aug.  1978. 

[99]  T.  f^vlidis,  Gramatical  and  graph  theoretical  analysis  of 
^^--^Tctures,  in  Graphic  languages,  Nake  and  Rosenfeld,  eds. 

Amsterdam,  North-Hoi  land,  1972. 

[100]  G.  Y.  Tang  and  T.  S.  Huang,  A  syntactic-semantic  approach  to 
image  understanding  and  creation,  IEEE  Trans.  Pattern  Analysis 
Machine  Intelligence,  Vol.  PAMI-1,  No.  2,  Apr.  1979. 
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A  syntactic  method  for  the  analysis  of  time-varying  image  patterns  is  proposed 
and  studied.  This  method  utilizes  translation  schema  to  model  the  time-varying 
properties  of  image  patterns.  A  syntactic  deformation  model  is  first  applied  to  trans¬ 
form  the  i-th  image  into  the  (i+l)-th  image  of  an  image  sequence.  Then  the  concept 
of  translation  in  formal  language  theory  is  used  as  a  mechanism  to  characterize  the 
dynamic  process  of  the  image  sequence.  A  formulation  of  stochastic  translation  is  also 
presented.  A  generalized  syntax-directed  tree  translation  model  is  proposed  to  handle 
h igh-d imens ional  patterns.  The  generalized  model  is  compared  with  the  conventional 
top-down  and  bottom-up  tree  translation  models. 

A  traffic  monitoring  problem  is  analyzed  using  the  proposed  tree  translation 
model.  Each  input  image  is  represented  as  a  tree  structure.  The  proposed  tree 
translation  model  is  used  to  model  the  variation  of  image  content  between  consecutive 
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images.  A  parsing  algorithm  for  tree  translation 
(vehicles)  in  each  pair  of  consecutive  images. 


is  applied  to  match  moving  objects 
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